AutoGPT 是面向自主智能体的开源平台,支持用自然语言或可视化流程创建、运行和管理能够持续执行任务的智能体。
关于 BuboGPT
BuboGPT BuboGPT: Enabling Visual Grounding in Multi-Modal LLMs Yang Zhao * , Zhijie Lin * , Daquan Zhou , Zilong Huang , Jiashi Feng , Bingyi Kang + , ▶ Bytedance Inc.    * Equal Contribution    + Project Lead arXiv Code Demo Dataset Model Video --> BuboGPT is an advanced Large Language Model (LLM) that incorporates multi-modal inputs including text, image and audio, with a unique ability to ground its responses to visual objects. It demonstrates remarkable chat abilities for arbitrary image-audio data understanding, whether aligned or unaligned. ▶ Bubo owls are well known for having strong vision and hearing abilities that help them thrive. --> --> --> --> --> Abstract LLMs have demonstrated remarkable abilities at interacting with humans through language, especially with the usage of instruction-following data. Recent advancements in LLMs, such as MiniGPT-4, LLaVA, and X-LLM, further enlarge their abilities by incorporating multi-modal inputs, including image, video, and speech. Despite their effectiveness at generating precise and detailed language understanding of the given modality signal, these LLMs give up the ability to ground specific parts of inputs, thus only constructing a coarse-grained mapping. However, explicit and informative correspondence between text and other modalities will not only improve the user experience but also help to expand the application scenario of multi-modal LLMs. BuboGPT Architecture . We build a multi-modal LLM, BuboGPT for multi-modal understanding including image, audio and text by learning a common semantic space and further explore the fine-grained relation between different visual objects and different modalities. Multimodal Instruct Data . We construct a high-quality multi-modal instruction-tuning dataset including fine-grained audio descriptions and cross-modal sound localization, and introduce both positive and negative image-audio pairs for semantic matching to facilitate the cross-modal understanding.. BuboGPT Architecture --> As the figure shown, we perform joint multi-modal understanding and chatting for text, vision and audio, which is achieved by learning a shared representation space that aligns well with pre-trained Vicuna . We also build an off-the-shelf visual grounding pipeline to explore the fine-grained relation between different visual objects and modalities. The framework of BuboGPT. [HuggingFace Dataset] .--> --> --> --> --> Data file name --> File Size --> Sample Siz
访问提示
本站仅提供官网信息与访问导航,不代表对该网站的产品、服务或交易作出担保。访问外部网站时,请核对网址,注意账号和财产安全。
同类型网站
查看分类百川智能发布的金融大模型,金融场景能力领先GPT-4o近20%
Build a Large Language Model (From Scratch) 是一款AI 大模型方向的 AI 工具,适合主流大语言模型与开源模型。
支持多 AI Agent 框架的可视化管理面板,支持 OpenClaw 和 Hermes Agent 双引擎,内置智能助手一键安装诊断