Video generation models (VGMs) offer a promising pathway for unified world modeling in robotics by integrating simulation, prediction, and manipulation. However, their practical application remains limited due to (1) slowgeneration speed, which limits real-time interaction, and (2) poor consistency between imagined videos and executable actions.
To address these challenges, we propose Manipulate in Dream (MinD), a hierarchical diffusion-based world model framework that employs a dual-system design for vision-language manipulation. MinD executes VGM at low frequencies to extract video prediction features, while leveraging a high-frequency diffusion policy for real-time interaction. This architecture enables low-latency, closed-loop control in manipulation with coherent visual guidance.
To better coordinate the two systems, we introduce a video-action diffusion matching module (DiffMatcher), with a novel co-training strategy that uses separate schedulers for each diffusion model. Specifically, we introduce a diffusion-forcing mechanism to DiffMatcher that aligns their intermediate representations during training, helping the fast action model better understand video-based predictions. Beyond manipulation, MinD also functions as a world simulator, reliably predicting task success or failure in latent space before execution. Trustworthy analysis further shows that VGMs can preemptively evaluate task feasibility and mitigate risks. Extensive experiments across multiple benchmarks demonstrate that MinD achieves state-of-the-art manipulation (63%+) in RL-Bench, advancing the frontier of unified world modeling in robotics.
MinD is a general-purpose multimodal world model for robotic manipulation that integrates visual imagination and action planning. Its core consists of a hierarchical diffusion-based framework with three components:
During inference, LoDiff-Visual forward-simulates noisy visual latents, which DiffMatcher transforms into aligned features to condition HiDiff-Policy for action generation. The system is trained with a dual-scheduler co-training strategy, optimizing a total objective that includes a video loss, an action loss, and a regularization loss for DiffMatcher (to enforce consistency between noisy and clean visual features), ensuring robust performance across asynchronous temporal scales and imperfect inputs.
Comparison of video generation result from LoDiff-Visual against real execution observation of HiDiff-Policy of RL-Bench and real-world Franka. |
We first evaluate MinD in the RL-Bench evaluation environment. This simulation platform is a comprehensive robot learning benchmark and environment with 7 tasks designed to advance research in vision-guided robot manipulation with a single-arm Franka Panda robot and a front-view camera. We compare MinD with existing VLA models, including CogACT, RoboMamba, RoboDreamer, and OpenVLA. For the model using Mamba or LLM as the backbone, we colored it with a green background. We use a yellow background for the VLA models with a video generation backbone, and a red background for our method.
Evaluation and comparison on RL-Bench tasks. All models are finetuned on the collected 1000 trajectories (including 100 trajectories for each 7 tasks and 300 more randomly sampled tasks). |
The results show that MinD outperforms all the existing VLA models, especially in tasks requiring complex temporal reasoning, such as "Sweep to Dustpan" (96%) and "Close Laptop Lid" (68%), highlighting the strong capability of video generation models as the foundational backbone for comprehensive visual-language manipulations. Besides, MinD also achieves the highest inference speed of 11.3 FPS, showcasing its superior efficiency.
We evaluate MinD with a Franka Research 3 Robot to perform 4 real-world tasks: 1) pick and place, 2) unplug the charger, 3) pour water, 4) wipe the whiteboard. We collected a dataset with 100 human demonstration trajectories via teleoperation using a SpaceMouse. As shown in the table below, our model achieves competitive performance across all tasks, with notable strengths in tasks requiring precise manipulation, such as pick and place(60%) and wiping the whiteboard (65%).
Real-world evaluation with the Franka robot across four tasks, each with 20 trials of random configurations. |
We evaluate each configuration based on video generation quality (FVD [30]) and success rate (SR) in task execution. The results highlight the impact of key components such as LDP, diffusion modules (LoDiff, DiffMatcher, HiDiff), and loss functions (Lvideo, Lsim, Laction) on performance. It also shows that large-scale video data pretraining and diffusion modules are vital for improving how well our MinD robot framework executes and generates videos. In short, using both video and action data, pretraining, and all the loss functions are key for best results in robot learning.
Ablation study results. SE denotes the state encoder, LDP represents large-scale data pretraining, A denotes action and V is video. |
We also conducted case study exploring how video generation models (VGMs) enhance the trustworthiness of world-model-based VLA by enabling risk assessment and outcome prediction for robotic tasks. We demonstrate that VGMs can predict both successful and failed executions, offering actionable insights for safer real-world deployment. While effective, future work should aim to improve motion prediction and incorporate richer multimodal inputs for more robust and reliable VLA.
The left panel shows the confusion matrix, highlighting prediction accuracy for task outcomes. The right panel visualizes a failing case (top) with trajectory misalignment and a successful case (bottom) with accurate prediction. |
@article{chi2025_2506.18897,
title={ MinD: Unified Visual Imagination and Control via Hierarchical World Models },
author={ Xiaowei Chi and Kuangzhi Ge and Jiaming Liu and Siyuan Zhou and Peidong Jia and Zichen He and Yuzhen Liu and
Tingguang Li and Lei Han and Sirui Han and Shanghang Zhang and Yike Guo },
journal={arXiv preprint arXiv:2506.18897},
year={ 2025 }
}