AnyMo arXiv ↗

AnyMoScaling Any-Modality Conditional Motion
Generation with Masked Modeling

Yiheng Li1,2Zhuo Li3,†Ruibing Hou1,‡Yingjie Chen3Hong Chang1,2Hao Liu3Shiguang Shan1,2

1Institute of Computing Technology, Chinese Academy of Sciences2University of Chinese Academy of Sciences3Independent researchers

† Project leader   ‡ Corresponding author

NeurIPS 2026

Paper arXiv Dataset BibTeX
Any modality. One motion model. Generate human motion from text, speech, music, trajectories — and their combinations.
3.2Mmotion sequences
5,048 hhuman motion data
~500 haligned audio subset

Abstract

Conditional human motion generation remains a fundamental challenge in computer vision and robotics. Despite significant progress, current methods are often constrained by fixed modality configurations and task-specific architectures, leaving cross-modal interactions and the scaling laws of multimodal-conditioned synthesis largely underexplored. A key bottleneck is the scarcity of large-scale modality-aligned motion data, limiting generalization across diverse control signals.

In this work, we introduce OmniHuMo, a large-scale, high-quality dataset comprising over 5,000 hours of motion and 3.2 million sequences with precisely aligned multimodal annotations (e.g., text, speech, music, and trajectory). Leveraging OmniHuMo, we propose AnyMo, a unified multimodal framework combining a Residual FSQ-based motion tokenizer with a scalable masked modeling transformer, enabling high-quality motion synthesis under arbitrary modality combinations. Extensive experiments show that AnyMo achieves high-fidelity synthesis while offering flexible control over both spatial and stylistic attributes.

Any-Modality Motion Generation

From an action description to a voice, a beat, or a path.
One framework brings diverse conditioning signals into motion.

Text prompts guide diverse actions and multi-step motion sequences.
Qualitative results from the paper · static motion trailsFigure S6 ↗

OmniHuMo Dataset

Large-scale motion. Multimodal annotations. A shared foundation.

Automatic data construction pipeline · Figure 2

From videos to
multimodal motion.

OmniHuMo turns diverse online videos into a corpus of 3.2 million human motion sequences, spanning 5,048 hours of movement.

Every sequence is paired with 1–3 text captions. An approximately 500-hour subset also includes temporally aligned speech or music, enabling learning across semantics, rhythm, and motion.

Explore OmniHuMo on Hugging Face ↗

BibTeX

@misc{li2026anymoscalinganymodalityconditional,
  title={AnyMo: Scaling Any-Modality Conditional Motion
         Generation with Masked Modeling},
  author={Yiheng Li and Zhuo Li and Ruibing Hou and
          Yingjie Chen and Hong Chang and Hao Liu and Shiguang Shan},
  year={2026},
  eprint={2605.29488},
  archivePrefix={arXiv},
  primaryClass={cs.CV},
  url={https://arxiv.org/abs/2605.29488}
}

arXiv citation. The final proceedings citation can be added when available.

Paper figure