Abstract:

Manipulation on industrial conveyors is inherently time-sensitive. Conveyor motion can shift the required grasp or placement location before the robot’s end effector reaches it. Existing approaches may require task-specific demonstrations, policy training, instance-specific object models, or configured conveyor motion, which can limit transfer to new objects and motion conditions. We introduce DYNAMIT, an instruction-conditioned, zero-shot framework for conveyor pick-and-place that operates without these requirements. The language instruction specifies the objects and receiving receptacles, while memory-based visual tracking preserves their identities in RGB-D observations. Depth measurements and track histories provide estimates of each instance’s position, dimensions, and velocity. A reachability-aware planner uses these estimates and robot proprioception to plan when and where each grasp or placement should occur, accounting for the end effector’s travel time. The planner updates this timing and location during execution, allowing the same pipeline to grasp moving objects and place objects into moving receptacles. We evaluate two conveyor tasks in simulation at different conveyor speeds and test transfer to industrial objects. DYNAMIT achieves the highest aggregate success rate and Q-score among the evaluated methods in each conveyor study. Evaluation on the Dynamic Object Manipulation (DOM) benchmark further demonstrates transfer to new scenes and task conditions without DOM demonstrations or policy training.