好网站,从官网开始
企业认领/联系与合作
官网导航 提交官网
J

JoyHallo官网

一个功能强大的数字人模型,专注于普通话和英语的音频驱动视频生成。它通过优化模型结构和数据集支持,实现了高效的跨语言生成能力,并在推理速度和准确性上表现出色

官网资料

网站名称
JoyHallo
所属分类
AI音频音乐
身份核实
尚未完成企业认领核实
最近更新
链接检测
尚未检测

关于 JoyHallo

JoyHallo More Research JoyHallo JoyType JoyVASA Citrus-V JoyHallo: Digital human model for Mandarin Sheng Shi ,  Xuyang Cao ,  Jun Zhao ,  Guoxin Wang JD Health International Inc. 1 zhaojun10@jd.com --> Paper arXiv Code HuggingFace Gradio Your browser does not support the video tag. Abstract In audio-driven video generation, creating Mandarin videos presents significant challenges. Collecting comprehensive Mandarin datasets is difficult, and the complex lip movements in Mandarin further complicate model training compared to English. In this study, we collected 29 hours of Mandarin speech video from JD Health International Inc. employees, resulting in the jdh-Hallo dataset. This dataset includes a diverse range of ages and speaking styles, encompassing both conversational and specialized medical topics. To adapt the JoyHallo model for Mandarin, we employed the Chinese wav2vec2 model for audio feature embedding. A semi-decoupled structure is proposed to capture inter-feature relationships among lip, expression, and pose features. This integration not only improves information utilization efficiency but also accelerates inference speed by 14.3%. Notably, JoyHallo maintains its strong ability to generate English videos, demonstrating excellent cross-language generation capabilities. Framework Improved aesthetics quality --> Figure 1. Pipeline of training and inferencing --> --> Videos Mandarin --> Mandarin Speech Mandarin Song English Speech English Song ← → --> Improved aesthetics quality User study results --> Table 2. Performance on less recognizable artistic fonts --> --> Comparison Improved aesthetics quality --> Table 1. Accuracy comparison in Mandarin Table 2. Accuracy comparison in English Table 3. Inference efficiency comparison BibTeX @misc{shi2024joyhallo, title={JoyHallo: Digital human model for Mandarin}, author={Sheng Shi and Xuyang Cao and Jun Zhao and Guoxin Wang}, year={2024}, eprint={2409.13268}, archivePrefix={arXiv}, primaryClass={cs.CV}, url={https://arxiv.org/abs/2409.13268}, } This page was built using the Academic Project Page Template which was adopted from the Nerfies project page.

访问提示

本站仅提供官网信息与访问导航,不代表对该网站的产品、服务或交易作出担保。访问外部网站时,请核对网址,注意账号和财产安全。

同类型网站 318

查看分类

AISong.Fun 是一个使用 AI 生成歌曲的平台,支持歌词和纯音乐创作,免费提供两次生成机会。

AssemblyAI是一个为开发者提供高精度语音转文本API及情感分析、内容摘要等高级语音智能功能的平台,帮助企业从音频和视频中提取文本洞察

让更多人找到你的官网

提交网站信息,建立清晰、完整的官网资料。

免费提交官网

咨询提交、推广、认领

客服微信二维码,长按识别或保存后使用微信扫一扫

扫一扫添加微信,沟通官网导航提交、认领、推广需求。

可长按识别,或保存图片后在微信扫一扫中识别。

添加时可备注“官网导航”,便于沟通您的企业需求。

保存二维码 打开原图