Abstract:

Robotic imitation learning requires policies that capture multimodal action distributions, run at real-time control rates, and exploit available sensors. Existing generative approaches, including diffusion, flow matching, and IMLE, satisfy these requirements only partially. We present PRISM, a single-pass multi-sensory policy built on a calibrated batch-global RS–IMLE objective. We show that the RS–IMLE rejection mechanism, originally designed for few-shot image synthesis, degenerates at policy-scale batch sizes, and introduce the calibration needed to restore effective rejection. PRISM couples a temporal multisensory encoder (RGB, depth, tactile, audio, proprioception) with a linear-attention Performer generator trained under this objective. On real-world loco-manipulation (Unitree Go2 + 7-DoF arm) and tabletop manipulation (UR5), PRISM improves success over diffusion, flow, and IMLE baselines by 10–25 pp while sustaining 30-50 Hz closed-loop control. On language-conditioned CALVIN ABCD, it improves success by about 31 percentage points over diffusion and about 24 pp over flow matching, while reducing trajectory jerk by 20x–70x.