AISong.Fun 是一个使用 AI 生成歌曲的平台,支持歌词和纯音乐创作,免费提供两次生成机会。
关于 Draw an Audio
Draw an Audio --> --> --> Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesis Qi Yang 1 , Binjie Mao 2 , Zili Wang 1 , Xing Nie 1 , Pengfei Gao 2 , Ying Guo 2 , Cheng Zhen 2 , Pengfei Yan 2 , Shiming Xiang 1 1 Institute of Automation, Chinese Academy of Sciences, 2 Meituan Paper arXiv Code (coming soon) Abstract Foley is a term commonly used in filmmaking, referring to the addition of daily sound effects to silent films or videos to enhance the auditory experience. Video-to-Audio (V2A), as a particular type of automatic foley task, presents inherent challenges related to audio-visual synchronization. These challenges encompass maintaining the content consistency between the input video and the generated audio, as well as the alignment of temporal and loudness properties within the video. To address these issues, we construct a controllable video-to-audio synthesis model, termed Draw an Audio, which supports multiple input instructions through drawn masks and loudness signals. To ensure content consistency between the synthesized audio and target video, we introduce the Mask-Attention Module (MAM), which employs masked video instruction to enable the model to focus on regions of interest. Additionally, we implement the Time-Loudness Module (TLM), which uses an auxiliary loudness signal to ensure the synthesis of sound that aligns with the video in both loudness and temporal dimensions. Furthermore, we have extended a large-scale V2A dataset, named VGGSound-Caption, by annotating caption prompts. Extensive experiments on challenging benchmarks across two large-scale V2A datasets verify Draw an Audio achieves the state-of-the-art. --> Visualization Vanilla Foley v.s. Draw an Audio. The traditional methods produce the entire audio only from video inputs, demonstrating limitations in controllability and flexibility. Draw an Audio, in contrast, offers a more appealing alternative that employs multiple instructions to produce high-quality synchronized audio and can produce mixed audio in multi-stages, thereby exhibiting greater practical application. Architecture The architecture of Draw an Audio, which incorporates a Latent Diffusion Model (LDM) as the foundational model, a Text Condition Model for text instruction, a Masked-Attention Module (MAM) for video instruction, a Time-Loudness Module (TLM) for signal instruction. Demos It is recommended to use earphones to hear the demos videos, raise the volume and zoom in the videos. unconditional
访问提示
本站仅提供官网信息与访问导航,不代表对该网站的产品、服务或交易作出担保。访问外部网站时,请核对网址,注意账号和财产安全。
同类型网站
查看分类AssemblyAI是一个为开发者提供高精度语音转文本API及情感分析、内容摘要等高级语音智能功能的平台,帮助企业从音频和视频中提取文本洞察
Meta推出的免费开源的AI语音和声音生成模型
听起来很棒!AI一键清除音频背景杂音