Robotics
Where hardware, software and machine intelligence come together.
Our research is interdisciplinary and focuses on sensing, planning, reasoning, and control of single and multi-agent systems, including both manipulation and mobile robots. We strive to develop algorithms and methods for factory automation, smart building and transportation applications using machine learning, computer vision, RF/optical sensing, wireless communications, control theory and signal processing. Key research themes include bin picking and object manipulation, sensing and mapping of indoor areas, coordinated control of robot swarms, as well as robot learning and simulation.
Quick Links
-
Researchers
Devesh K.
Jha
Diego
Romeres
Daniel N.
Nikovski
Stefano
Di Cairano
Arvind
Raghunathan
Siddarth
Jain
William S.
Yerazunis
Radu
Corcodel
Yebin
Wang
Toshiaki
Koike-Akino
Yuki
Shirai
Abraham P.
Vinod
Avishai
Weiss
Tim K.
Marks
Scott A.
Bortoff
Chiori
Hori
Ye
Wang
Anoop
Cherian
Jonathan
Le Roux
Matthew
Brand
Philip V.
Orlik
Alexander
Schperberg
Bingnan
Wang
Purnanand
Elango
Abraham
Goldsmith
Jianlin
Guo
Sameer
Khurana
Jing
Liu
Hassan
Mansour
Pedro
Miraldo
Saviz
Mowlavi
Anthony
Vetro
James
Queeney
-
Awards
-
AWARD University of Padua and MERL team wins the AI Olympics with RealAIGym competition at IROS24 Date: October 17, 2024
Awarded to: Niccolò Turcato, Alberto Dalla Libera, Giulio Giacomuzzo, Ruggero Carli, Diego Romeres
MERL Contact: Diego Romeres
Research Areas: Artificial Intelligence, Dynamical Systems, Machine Learning, RoboticsBrief- The team composed of the control group at the University of Padua and MERL's Optimization and Robotic team ranked 1st out of the 4 finalist teams that arrived to the 2nd AI Olympics with RealAIGym competition at IROS 24, which focused on control of under-actuated robots. The team was composed by Niccolò Turcato, Alberto Dalla Libera, Giulio Giacomuzzo, Ruggero Carli and Diego Romeres. The competition was organized by the German Research Center for Artificial Intelligence (DFKI), Technical University of Darmstadt and Chalmers University of Technology.
The competition and award ceremony was hosted by IEEE International Conference on Intelligent Robots and Systems (IROS) on October 17, 2024 in Abu Dhabi, UAE. Diego Romeres presented the team's method, based on a model-based reinforcement learning algorithm called MC-PILCO.
- The team composed of the control group at the University of Padua and MERL's Optimization and Robotic team ranked 1st out of the 4 finalist teams that arrived to the 2nd AI Olympics with RealAIGym competition at IROS 24, which focused on control of under-actuated robots. The team was composed by Niccolò Turcato, Alberto Dalla Libera, Giulio Giacomuzzo, Ruggero Carli and Diego Romeres. The competition was organized by the German Research Center for Artificial Intelligence (DFKI), Technical University of Darmstadt and Chalmers University of Technology.
-
AWARD Honorable Mention Award at NeurIPS 23 Instruction Workshop Date: December 15, 2023
Awarded to: Lingfeng Sun, Devesh K. Jha, Chiori Hori, Siddharth Jain, Radu Corcodel, Xinghao Zhu, Masayoshi Tomizuka and Diego Romeres
MERL Contacts: Radu Corcodel; Chiori Hori; Siddarth Jain; Devesh K. Jha; Diego Romeres
Research Areas: Artificial Intelligence, Machine Learning, RoboticsBrief- MERL Researchers received an "Honorable Mention award" at the Workshop on Instruction Tuning and Instruction Following at the NeurIPS 2023 conference in New Orleans. The workshop was on the topic of instruction tuning and Instruction following for Large Language Models (LLMs). MERL researchers presented their work on interactive planning using LLMs for partially observable robotic tasks during the oral presentation session at the workshop.
-
AWARD Joint University of Padua-MERL team wins Challenge 'AI Olympics With RealAIGym' Date: August 25, 2023
Awarded to: Alberto Dalla Libera, Niccolo' Turcato, Giulio Giacomuzzo, Ruggero Carli, Diego Romeres
MERL Contact: Diego Romeres
Research Areas: Artificial Intelligence, Machine Learning, RoboticsBrief- A joint team consisting of members of University of Padua and MERL ranked 1st in the IJCAI2023 Challenge "Al Olympics With RealAlGym: Is Al Ready for Athletic Intelligence in the Real World?". The team was composed by MERL researcher Diego Romeres and a team from University Padua (UniPD) consisting of Alberto Dalla Libera, Ph.D., Ph.D. Candidates: Niccolò Turcato, Giulio Giacomuzzo and Prof. Ruggero Carli from University of Padua.
The International Joint Conference on Artificial Intelligence (IJCAI) is a premier gathering for AI researchers and organizes several competitions. This year the competition CC7 "AI Olympics With RealAIGym: Is AI Ready for Athletic Intelligence in the Real World?" consisted of two stages: simulation and real-robot experiments on two under-actuated robotic systems. The two robotics systems were treated as separate tracks and one final winner was selected for each track based on specific performance criteria in the control tasks.
The UniPD-MERL team competed and won in both tracks. The team's system made strong use of a Model-based Reinforcement Learning algorithm called (MC-PILCO) that we recently published in the journal IEEE Transaction on Robotics.
- A joint team consisting of members of University of Padua and MERL ranked 1st in the IJCAI2023 Challenge "Al Olympics With RealAlGym: Is Al Ready for Athletic Intelligence in the Real World?". The team was composed by MERL researcher Diego Romeres and a team from University Padua (UniPD) consisting of Alberto Dalla Libera, Ph.D., Ph.D. Candidates: Niccolò Turcato, Giulio Giacomuzzo and Prof. Ruggero Carli from University of Padua.
See All Awards for Robotics -
-
News & Events
-
NEWS MERL researchers present 7 papers at CDC 2024 Date: December 16, 2024 - December 19, 2024
Where: Milan, Italy
MERL Contacts: Ankush Chakrabarty; Vedang M. Deshpande; Stefano Di Cairano; James Queeney; Abraham P. Vinod; Avishai Weiss; Gordon Wichern
Research Areas: Artificial Intelligence, Control, Dynamical Systems, Machine Learning, Multi-Physical Modeling, Optimization, RoboticsBrief- MERL researchers presented 7 papers at the recently concluded Conference on Decision and Control (CDC) 2024 in Milan, Italy. The papers covered a wide range of topics including safety shielding for stochastic model predictive control, reinforcement learning using expert observations, physics-constrained meta learning for positioning, variational-Bayes Kalman filtering, Bayesian measurement masks for GNSS positioning, divert-feasible lunar landing, and centering and stochastic control using constrained zonotopes.
As a sponsor of the conference, MERL maintained a booth for open discussions with researchers and students, and hosted a special session to discuss highlights of MERL research and work philosophy.
In addition, Ankush Chakrabarty (Principal Research Scientist, Multiphysical Systems Team) was an invited speaker in the pre-conference Workshop on "Learning Dynamics From Data" where he gave a talk on few-shot meta-learning for black-box identification using data from similar systems.
- MERL researchers presented 7 papers at the recently concluded Conference on Decision and Control (CDC) 2024 in Milan, Italy. The papers covered a wide range of topics including safety shielding for stochastic model predictive control, reinforcement learning using expert observations, physics-constrained meta learning for positioning, variational-Bayes Kalman filtering, Bayesian measurement masks for GNSS positioning, divert-feasible lunar landing, and centering and stochastic control using constrained zonotopes.
-
NEWS MERL Researchers to Present 2 Conference and 11 Workshop Papers at NeurIPS 2024 Date: December 10, 2024 - December 15, 2024
Where: Advances in Neural Processing Systems (NeurIPS)
MERL Contacts: Petros T. Boufounos; Matthew Brand; Ankush Chakrabarty; Anoop Cherian; François Germain; Toshiaki Koike-Akino; Christopher R. Laughman; Jonathan Le Roux; Jing Liu; Suhas Lohit; Tim K. Marks; Yoshiki Masuyama; Kieran Parsons; Kuan-Chuan Peng; Diego Romeres; Pu (Perry) Wang; Ye Wang; Gordon Wichern
Research Areas: Artificial Intelligence, Communications, Computational Sensing, Computer Vision, Control, Data Analytics, Dynamical Systems, Machine Learning, Multi-Physical Modeling, Optimization, Robotics, Signal Processing, Speech & Audio, Human-Computer Interaction, Information SecurityBrief- MERL researchers will attend and present the following papers at the 2024 Advances in Neural Processing Systems (NeurIPS) Conference and Workshops.
1. "RETR: Multi-View Radar Detection Transformer for Indoor Perception" by Ryoma Yataka (Mitsubishi Electric), Adriano Cardace (Bologna University), Perry Wang (Mitsubishi Electric Research Laboratories), Petros Boufounos (Mitsubishi Electric Research Laboratories), Ryuhei Takahashi (Mitsubishi Electric). Main Conference. https://neurips.cc/virtual/2024/poster/95530
2. "Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiads" by Anoop Cherian (Mitsubishi Electric Research Laboratories), Kuan-Chuan Peng (Mitsubishi Electric Research Laboratories), Suhas Lohit (Mitsubishi Electric Research Laboratories), Joanna Matthiesen (Math Kangaroo USA), Kevin Smith (Massachusetts Institute of Technology), Josh Tenenbaum (Massachusetts Institute of Technology). Main Conference, Datasets and Benchmarks track. https://neurips.cc/virtual/2024/poster/97639
3. "Probabilistic Forecasting for Building Energy Systems: Are Time-Series Foundation Models The Answer?" by Young-Jin Park (Massachusetts Institute of Technology), Jing Liu (Mitsubishi Electric Research Laboratories), François G Germain (Mitsubishi Electric Research Laboratories), Ye Wang (Mitsubishi Electric Research Laboratories), Toshiaki Koike-Akino (Mitsubishi Electric Research Laboratories), Gordon Wichern (Mitsubishi Electric Research Laboratories), Navid Azizan (Massachusetts Institute of Technology), Christopher R. Laughman (Mitsubishi Electric Research Laboratories), Ankush Chakrabarty (Mitsubishi Electric Research Laboratories). Time Series in the Age of Large Models Workshop.
4. "Forget to Flourish: Leveraging Model-Unlearning on Pretrained Language Models for Privacy Leakage" by Md Rafi Ur Rashid (Penn State University), Jing Liu (Mitsubishi Electric Research Laboratories), Toshiaki Koike-Akino (Mitsubishi Electric Research Laboratories), Shagufta Mehnaz (Penn State University), Ye Wang (Mitsubishi Electric Research Laboratories). Workshop on Red Teaming GenAI: What Can We Learn from Adversaries?
5. "Spatially-Aware Losses for Enhanced Neural Acoustic Fields" by Christopher Ick (New York University), Gordon Wichern (Mitsubishi Electric Research Laboratories), Yoshiki Masuyama (Mitsubishi Electric Research Laboratories), François G Germain (Mitsubishi Electric Research Laboratories), Jonathan Le Roux (Mitsubishi Electric Research Laboratories). Audio Imagination Workshop.
6. "FV-NeRV: Neural Compression for Free Viewpoint Videos" by Sorachi Kato (Osaka University), Takuya Fujihashi (Osaka University), Toshiaki Koike-Akino (Mitsubishi Electric Research Laboratories), Takashi Watanabe (Osaka University). Machine Learning and Compression Workshop.
7. "GPT Sonography: Hand Gesture Decoding from Forearm Ultrasound Images via VLM" by Keshav Bimbraw (Worcester Polytechnic Institute), Ye Wang (Mitsubishi Electric Research Laboratories), Jing Liu (Mitsubishi Electric Research Laboratories), Toshiaki Koike-Akino (Mitsubishi Electric Research Laboratories). AIM-FM: Advancements In Medical Foundation Models: Explainability, Robustness, Security, and Beyond Workshop.
8. "Smoothed Embeddings for Robust Language Models" by Hase Ryo (Mitsubishi Electric), Md Rafi Ur Rashid (Penn State University), Ashley Lewis (Ohio State University), Jing Liu (Mitsubishi Electric Research Laboratories), Toshiaki Koike-Akino (Mitsubishi Electric Research Laboratories), Kieran Parsons (Mitsubishi Electric Research Laboratories), Ye Wang (Mitsubishi Electric Research Laboratories). Safe Generative AI Workshop.
9. "Slaying the HyDRA: Parameter-Efficient Hyper Networks with Low-Displacement Rank Adaptation" by Xiangyu Chen (University of Kansas), Ye Wang (Mitsubishi Electric Research Laboratories), Matthew Brand (Mitsubishi Electric Research Laboratories), Pu Wang (Mitsubishi Electric Research Laboratories), Jing Liu (Mitsubishi Electric Research Laboratories), Toshiaki Koike-Akino (Mitsubishi Electric Research Laboratories). Workshop on Adaptive Foundation Models.
10. "Preference-based Multi-Objective Bayesian Optimization with Gradients" by Joshua Hang Sai Ip (University of California Berkeley), Ankush Chakrabarty (Mitsubishi Electric Research Laboratories), Ali Mesbah (University of California Berkeley), Diego Romeres (Mitsubishi Electric Research Laboratories). Workshop on Bayesian Decision-Making and Uncertainty. Lightning talk spotlight.
11. "TR-BEACON: Shedding Light on Efficient Behavior Discovery in High-Dimensions with Trust-Region-based Bayesian Novelty Search" by Wei-Ting Tang (Ohio State University), Ankush Chakrabarty (Mitsubishi Electric Research Laboratories), Joel A. Paulson (Ohio State University). Workshop on Bayesian Decision-Making and Uncertainty.
12. "MEL-PETs Joint-Context Attack for the NeurIPS 2024 LLM Privacy Challenge Red Team Track" by Ye Wang (Mitsubishi Electric Research Laboratories), Tsunato Nakai (Mitsubishi Electric), Jing Liu (Mitsubishi Electric Research Laboratories), Toshiaki Koike-Akino (Mitsubishi Electric Research Laboratories), Kento Oonishi (Mitsubishi Electric), Takuya Higashi (Mitsubishi Electric). LLM Privacy Challenge. Special Award for Practical Attack.
13. "MEL-PETs Defense for the NeurIPS 2024 LLM Privacy Challenge Blue Team Track" by Jing Liu (Mitsubishi Electric Research Laboratories), Ye Wang (Mitsubishi Electric Research Laboratories), Toshiaki Koike-Akino (Mitsubishi Electric Research Laboratories), Tsunato Nakai (Mitsubishi Electric), Kento Oonishi (Mitsubishi Electric), Takuya Higashi (Mitsubishi Electric). LLM Privacy Challenge. Won 3rd Place Award.
MERL members also contributed to the organization of the Multimodal Algorithmic Reasoning (MAR) Workshop (https://marworkshop.github.io/neurips24/). Organizers: Anoop Cherian (Mitsubishi Electric Research Laboratories), Kuan-Chuan Peng (Mitsubishi Electric Research Laboratories), Suhas Lohit (Mitsubishi Electric Research Laboratories), Honglu Zhou (Salesforce Research), Kevin Smith (Massachusetts Institute of Technology), Tim K. Marks (Mitsubishi Electric Research Laboratories), Juan Carlos Niebles (Salesforce AI Research), Petar Veličković (Google DeepMind).
- MERL researchers will attend and present the following papers at the 2024 Advances in Neural Processing Systems (NeurIPS) Conference and Workshops.
See All News & Events for Robotics -
-
Internships
-
OR0088: Internship - Robot Learning
MERL is looking for a highly motivated and qualified PhD student in the areas of machine learning and robotics, to participate in research on advanced algorithms for learning control of robots and other mechanisms. Solid background and hands-on experience with various machine learning algorithms is expected, and in particular with deep learning algorithms for image processing and object detection. Exposure to deep reinforcement learning and/or learning from demonstration is highly desirable. Familiarity with the use of machine learning algorithms for system identification of mechanical systems would be a plus, along with background in other areas of automatic control. Solid experimental skills and hands-on experience in coding in Python, PyTorch, and OpenCV are required for the position. Some experience with ROS2 and familiarity with classical mechanics and computational physics engines would be helpful, but is not required. The position will provide opportunities for exploring fundamental problems in incremental learning in humans and machines, leading to publishable results. The duration of the internship is 3 to 5 months, with a flexible starting date.
Required Specific Experience
- Python, PyTorch, OpenCV
-
CV0064: Internship - Robust Estimation for Computer Vision
MERL is looking for a self-motivated graduate student to work on robust estimation in Computer Vision. Based on the candidate’s interests, the intern can work on a variety of topics such as (but not limited to) camera pose estimation, 3D registration, camera calibration, pose-graph optimization, and transformation averaging. The ideal candidate would be a PhD student with a strong background in 3D computer vision, RANSAC, and graduated non-convexity algorithms, and good programming skills in C/C++ and/or Python. The candidate must have published at least one paper in a top-tier computer vision, machine learning, or robotics venue, such as CVPR, ECCV, ICCV, NeurIPS, ICRA, or IROS. The intern will collaborate with MERL researchers to derive and implement new algorithms for V-SLAM, conduct experiments, and report findings. A submission to a top-tier conference is expected. The duration of the internship and start date are flexible.
Required Specific Experience
- Experience with 3D computer vision, RANSAC, or graduated non-convexity algorithms for computer vision.
-
EA0065: Internship - Planning and Control of Mobile Manipulators
MERL is seeking a highly motivated and qualified individual to conduct research in safe/robust whole-body motion planning and control of mobile manipulators. The ideal candidate should demonstrate solid background and track record of publications in the areas of robotic dynamics, motion planning, and control. Strong C++ and Python coding skills, knowledge of robotic software such as Pinocchio/Pybullet/MuJoCo, and optimization tools such as CasADi/PyTorch are a necessity. Ph.D. students in mechanical engineering, robotics, computer science, and electrical engineering are encouraged to apply. Start date for this internship is flexible and the duration is about 3 months.
Required Specific Experience
- Solid background and track record of conducting innovative research in the dynamic modeling, motion planning, and control of robotic systems.
- Experience with C++/Python, Pinocchio, Pybullet, MuJoCo, CasADi, PyTorch.
See All Internships for Robotics -
-
Openings
-
CA0093: Research Scientist - Control for Autonomous Systems
-
CV0124: Postdoctoral Research Fellow - 3D Computer Vision
See All Openings at MERL -
-
Recent Publications
- "Invariant Set Planning for Quadrotors: Design, Analysis, Experiments", IEEE Transactions on Control Systems Technology, January 2025.BibTeX TR2025-010 PDF
- @article{Greiff2025jan,
- author = {Greiff, Marcus and Sinhmar, Himani and Weiss, Avishai and Berntorp, Karl and Di Cairano, Stefano}},
- title = {Invariant Set Planning for Quadrotors: Design, Analysis, Experiments},
- journal = {IEEE Transactions on Control Systems Technology},
- year = 2025,
- month = jan,
- url = {https://www.merl.com/publications/TR2025-010}
- }
, - "Continuous-Time Successive Convexification for Passively-Safe Six-Degree-of-Freedom Powered-Descent Guidance", AIAA SciTech, January 2025.BibTeX TR2025-008 PDF
- @inproceedings{Elango2025jan,
- author = {Elango, Purnanand and Vinod, Abraham P. and Di Cairano, Stefano and Weiss, Avishai}},
- title = {Continuous-Time Successive Convexification for Passively-Safe Six-Degree-of-Freedom Powered-Descent Guidance},
- booktitle = {AIAA SciTech},
- year = 2025,
- month = jan,
- url = {https://www.merl.com/publications/TR2025-008}
- }
, - "Chance-Constrained Optimization for Contact-rich Systems using Mixed Integer Programming", Nonlinear Analysis: Hybrid Systems, DOI: 10.1016/j.nahs.2024.101466, Vol. 52, December 2024.BibTeX TR2024-008 PDF
- @article{Shirai2024dec,
- author = {Shirai, Yuki and Jha, Devesh K. and Raghunathan, Arvind and Romeres, Diego},
- title = {Chance-Constrained Optimization for Contact-rich Systems using Mixed Integer Programming},
- journal = {Nonlinear Analysis: Hybrid Systems},
- year = 2024,
- volume = 52,
- month = dec,
- doi = {10.1016/j.nahs.2024.101466},
- issn = {1751-570X},
- url = {https://www.merl.com/publications/TR2024-008}
- }
, - "Learning Time-Optimal Control of Gantry Cranes", International Conference on Machine Learning and Applications (ICMLA), December 2024.BibTeX TR2024-181 PDF
- @inproceedings{Zhong2024dec,
- author = {{Zhong, Junmin and Nikovski, Daniel N. and Yerazunis, William S. and Ando, Taishi}},
- title = {Learning Time-Optimal Control of Gantry Cranes},
- booktitle = {International Conference on Machine Learning and Applications (ICMLA)},
- year = 2024,
- month = dec,
- url = {https://www.merl.com/publications/TR2024-181}
- }
, - "Chance-Constrained Information-Theoretic Stochastic Model Predictive Control with Safety Shielding", IEEE Conference on Decision and Control (CDC), December 2024.BibTeX TR2024-179 PDF
- @inproceedings{Yin2024dec,
- author = {Yin, Ji and Tsiotras, Panagiotis and Berntorp, Karl}},
- title = {Chance-Constrained Information-Theoretic Stochastic Model Predictive Control with Safety Shielding},
- booktitle = {IEEE Conference on Decision and Control (CDC)},
- year = 2024,
- month = dec,
- url = {https://www.merl.com/publications/TR2024-179}
- }
, - "Memory-Based Learning of Global Control Policies from Local Controllers", 21st International Conference on Informatics in Control, Automation and Robotics (ICINCO'24), November 2024.BibTeX TR2024-158 PDF
- @inproceedings{Nikovski2024nov,
- author = {{Nikovski, Daniel N. and Zhong, Junmin and Yerazunis, William S.}},
- title = {Memory-Based Learning of Global Control Policies from Local Controllers},
- booktitle = {21st International Conference on Informatics in Control, Automation and Robotics (ICINCO'24)},
- year = 2024,
- month = nov,
- url = {https://www.merl.com/publications/TR2024-158}
- }
, - "Open Human-Robot Collaboration Systems (OHRCS): A Research Perspective", IEEE International Conference on Cognitive Machine Intelligence (CogML 2024), October 2024.BibTeX TR2024-150 PDF
- @inproceedings{Suresh2024nov,
- author = {{Suresh, Prasanth and Romeres, Diego and Dosh, Prashant and Jain, Siddarth}},
- title = {Open Human-Robot Collaboration Systems (OHRCS): A Research Perspective},
- booktitle = {IEEE International Conference on Cognitive Machine Intelligence (CogML 2024)},
- year = 2024,
- month = oct,
- url = {https://www.merl.com/publications/TR2024-150}
- }
, - "Insert-One: One-Shot Robust Visual-Force Servoing for Novel Object Insertion with 6-DoF Tracking", 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2024), October 2024.BibTeX TR2024-137 PDF
- @inproceedings{Chang2024oct,
- author = {Chang, Haonan and Boularias, Abdeslam and Jain, Siddarth}},
- title = {Insert-One: One-Shot Robust Visual-Force Servoing for Novel Object Insertion with 6-DoF Tracking},
- booktitle = {2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2024)},
- year = 2024,
- month = oct,
- url = {https://www.merl.com/publications/TR2024-137}
- }
,
- "Invariant Set Planning for Quadrotors: Design, Analysis, Experiments", IEEE Transactions on Control Systems Technology, January 2025.
-
Videos
-
Software & Data Downloads
-
Lagrangian Inspired Polynomial for Robot Inverse Dynamics -
Monte Carlo Probabilistic Inference for Learning COntrol -
Python-based Robotic Control & Optimization Package -
Context-Aware Zero Shot Learning -
Online Feature Extractor Network -
Quasi-Newton Trust Region Policy Optimization -
Circular Maze Environment
-