AISong.Fun 是一个使用 AI 生成歌曲的平台,支持歌词和纯音乐创作,免费提供两次生成机会。
关于 JoyHallo
JoyHallo More Research JoyHallo JoyType JoyVASA Citrus-V JoyHallo: Digital human model for Mandarin Sheng Shi , Xuyang Cao , Jun Zhao , Guoxin Wang JD Health International Inc. 1 zhaojun10@jd.com --> Paper arXiv Code HuggingFace Gradio Your browser does not support the video tag. Abstract In audio-driven video generation, creating Mandarin videos presents significant challenges. Collecting comprehensive Mandarin datasets is difficult, and the complex lip movements in Mandarin further complicate model training compared to English. In this study, we collected 29 hours of Mandarin speech video from JD Health International Inc. employees, resulting in the jdh-Hallo dataset. This dataset includes a diverse range of ages and speaking styles, encompassing both conversational and specialized medical topics. To adapt the JoyHallo model for Mandarin, we employed the Chinese wav2vec2 model for audio feature embedding. A semi-decoupled structure is proposed to capture inter-feature relationships among lip, expression, and pose features. This integration not only improves information utilization efficiency but also accelerates inference speed by 14.3%. Notably, JoyHallo maintains its strong ability to generate English videos, demonstrating excellent cross-language generation capabilities. Framework Improved aesthetics quality --> Figure 1. Pipeline of training and inferencing --> --> Videos Mandarin --> Mandarin Speech Mandarin Song English Speech English Song ← → --> Improved aesthetics quality User study results --> Table 2. Performance on less recognizable artistic fonts --> --> Comparison Improved aesthetics quality --> Table 1. Accuracy comparison in Mandarin Table 2. Accuracy comparison in English Table 3. Inference efficiency comparison BibTeX @misc{shi2024joyhallo, title={JoyHallo: Digital human model for Mandarin}, author={Sheng Shi and Xuyang Cao and Jun Zhao and Guoxin Wang}, year={2024}, eprint={2409.13268}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2409.13268}, } This page was built using the Academic Project Page Template which was adopted from the Nerfies project page.
访问提示
本站仅提供官网信息与访问导航,不代表对该网站的产品、服务或交易作出担保。访问外部网站时,请核对网址,注意账号和财产安全。
同类型网站
查看分类AssemblyAI是一个为开发者提供高精度语音转文本API及情感分析、内容摘要等高级语音智能功能的平台,帮助企业从音频和视频中提取文本洞察
Meta推出的免费开源的AI语音和声音生成模型
听起来很棒!AI一键清除音频背景杂音