MagicPose4D: Crafting Articulated Models
with Appearance and Motion Control

Hao Zhang1*   Di Chang2*   Fang Li1   Mohammad Soleymani2   Narendra Ahuja1
1University of Illinois Urbana-Champaign      2University of Southern California
*Equal Contribution    
Paper arXiv Code

We introduce MagicPose4D, a novel framework for 4D generation providing more accurate and customizable 4D motion retargeting. We propose a dual-phase reconstruction process that initially uses accurate 2D and pseudo 3D supervision without skeleton constraints, and subsequently refines the model with skeleton constraints to ensure physical plausibility. We incorporate a novel Global-Local Chamfer loss function that aligns the overall distribution of mesh vertices with the supervision and maintains part-level alignment without additional annotations. Our method enables cross-category motion transfer using a kinematic-chain-based skeleton, ensuring smooth transitions between frames through dynamic rigidity and achieving robust generalization without the need for additional training.


empty

Video

Method

Interpolate start reference image.


MagicPose4D accepts two distinct types of input prompts: (i) appearance prompts and (ii) motion prompts. Both images and textual descriptions can function as appearance prompts, delineating the desired object and its visual characteristics. In a departure from existing approaches, MagicPose4D enables users to specify precise motions and trajectories by providing a video/mesh sequence that represents the anticipated movement. MagicPose4D comprises three critical components: (i) the 4D Reconstruction module, (ii) the Cross-Category Motion Transfer module, and (iii) the Image-to-3D Generation module. Each module is tailored to facilitate distinct aspects of dynamic modeling, enabling adaptive 4D reconstructions that align with user-defined specifications.

Multi-View 4D Generation

Animal

Humanoid

Comparison to Other Pose Transfer Methods

Animals

Reference Motion Ours 3D-CoreNet X-DualNet




Humanoid

Reference Motion Ours 3D-CoreNet X-DualNet

Comparison to Image-to-4D Generation Method

"Hip-Hop Dancing"

"Running like a rabit"

"Attacking with head"

BibTeX

@misc{zhang2024magicpose4d,
      title={MagicPose4D: Crafting Articulated Models with Appearance and Motion Control}, 
      author={Hao Zhang and Di Chang and Fang Li and Mohammad Soleymani and Narendra Ahuja},
      year={2024},
      eprint={2405.14017},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}