NVIDIA unveils Cosmos-H-Dreams, a real-time, action-conditioned generative simulator for surgical robotics designed to accelerate evaluation and training of vision-language-action policies in surgical systems.

The system is a real-time surgical world model capable of generating the operating field at about 160 frames per second on a single RTX PRO 6000 GPU, hinting at potential real-time guidance for surgeons or robot policies.

The Medical Physics Simulation framework underpins the model by combining a learned visual dynamics model with traditional solvers for contact and friction, with fast-version accuracy and long-term fidelity yet to be proven and requiring closed-loop testing against metrics like tool-tip pose accuracy and gripper fidelity.

The model is a distilled student of a slower teacher model and uses self-forcing distillation, with FlashDreams handling streaming inference and pre-caching to avoid repeated re-planning.

Training includes failure and out-of-distribution episodes so the simulator can score policies by reproducing consequences of poor actions, not just ideal demonstrations.

Accessibility is favorable for researchers: weights under an open model license, serving code under Apache 2.0, modest hardware requirements (12 GB GPU, Linux, modern driver), though a technical report is not yet published.

The released checkpoint demonstrates tabletop suturing on the da Vinci Research Kit at 288 by 512 pixels, effectively 10 Hz, indicating a bench task rather than full-body surgery.

World foundation models are leveraged to learn visual dynamics from synchronized video and robot kinematics, avoiding manual authoring of every object and interaction in complex surgical scenes.

Demonstrations are for research and demonstration purposes only, not part of cleared clinical systems and not intended for clinical decision-making.

Compared with traditional simulators, Cosmos-H-Dreams tackles tissue deformation, instrument interactions, sutures, needles, smoke, specular surfaces, and occlusions to enable safer, faster, and more scalable evaluation and data generation.

NVIDIA collaborated with CMR Surgical and Cambridge Consultants to connect the model to the Versius robotic platform for live demonstrations, with anonymized Versius procedure data contributing to the Open-H Embodiment dataset used in pretraining.

The simulator supports interactive, closed-loop control by a human operator or a learned policy, running on a single RTX PRO 6000 GPU.