AI Student Pack 是一款AI 教育与教学方向的 AI 工具,适合AI 教学备课、个性化学习与教育管理工具。
关于 ActAnywhere
ActAnywhere ActAnywhere Subject-Aware Video Background Generation Boxiao Pan 1,2 Zhan Xu 2 Chun-Hao Paul Huang 2 Krishna Kumar Singh 2 Yang Zhou 2 Leonidas J. Guibas 1 Jimei Yang 3 1 Stanford University 2 Adobe Research 3 Runway NeurIPS 2024 [arXiv] Subject segmentation sequence + Image of a background → Subject-aware video background! Abstract We study a novel problem to automatically generate video background that tailors to foreground subject motion. It is an important problem for the movie industry and visual effects community, which traditionally requires tedious manual efforts to solve. To this end, we propose ActAnywhere , a video diffusion model that takes as input a sequence of foreground subject segmentation together with an image of a novel background, and generates a video of the subject interacting in this background. We train our model on a large-scale dataset of 2.4M videos of human-scene interactions. Through extensive evaluation, we show that our model produces videos with realistic foreground-background interaction while strictly following the guidance of the condition image. Our model generalizes to diverse scenarios including non-human subjects, gaming and animation clips, as well as videos with multiple moving subjects. Both quantitative and qualitative comparisons demonstrate that our model significantly outperforms existing methods, which fail to accomplish the studied task. Method During training, we take a randomly sampled frame from the training video to condition the denoising process. At test time, the condition can be either a composited frame of the subject with a novel background, or a background-only image. Results Click on dropdowns to view different categories. Videos should play automatically and in a loop. We used Adobe Firefly to generate the composited frames shown here. Hover mouse over them to see the corresponding text prompts, which are either produced from ChatGPT 4 or manually written. Video background generation with composited frame conditioning Original video (not used as model input) Segmentation Mallard wandering around a firepit. Condition Output Original video (not used as model input) Segmentation A man folding bed sheets. Condition Output Original video (not used as model input) Segmentation Purple tie-dye jogger runs in serene park, mist over lake. Condition Output Original video (not used as model input) Segmentation A woman is water-skiing. Condition Output Original video (not used as model input) Seg
访问提示
本站仅提供官网信息与访问导航,不代表对该网站的产品、服务或交易作出担保。访问外部网站时,请核对网址,注意账号和财产安全。
同类型网站
查看分类BlogcastTM是一个文本转语音的工具,允许用户创建播客、视频、电子学习课程的音频和音。
Khan Academy (Khanmigo) 是一款面向教育学习场景的智能辅导工具,适合课程制作、学习辅导与教学管理。
MIT OpenCourseWare - Artificial Intelligence 是一款AI 学习方向的 AI 工具,适合AI 学习资源与课程。