Robotics
Where hardware, software and machine intelligence come together.
Our research is interdisciplinary and focuses on sensing, planning, reasoning, and control of single and multi-agent systems, including both manipulation and mobile robots. We strive to develop algorithms and methods for factory automation, smart building and transportation applications using machine learning, computer vision, RF/optical sensing, wireless communications, control theory and signal processing. Key research themes include bin picking and object manipulation, sensing and mapping of indoor areas, coordinated control of robot swarms, as well as robot learning and simulation.
Quick Links
-
Researchers
Devesh K.
Jha
Diego
Romeres
Daniel N.
Nikovski
Arvind
Raghunathan
Stefano
Di Cairano
Siddarth
Jain
Radu
Corcodel
Yebin
Wang
William S.
Yerazunis
Karl
Berntorp
Toshiaki
Koike-Akino
Yuki
Shirai
Mouhacine
Benosman
Tim K.
Marks
Abraham P.
Vinod
Avishai
Weiss
Scott A.
Bortoff
Chiori
Hori
Jonathan
Le Roux
Ye
Wang
Matthew
Brand
Anoop
Cherian
Philip V.
Orlik
Bingnan
Wang
Abraham
Goldsmith
Jianlin
Guo
Sameer
Khurana
Hassan
Mansour
Saviz
Mowlavi
Anthony
Vetro
Pedro
Miraldo
James
Queeney
-
Awards
-
AWARD Honorable Mention Award at NeurIPS 23 Instruction Workshop Date: December 15, 2023
Awarded to: Lingfeng Sun, Devesh K. Jha, Chiori Hori, Siddharth Jain, Radu Corcodel, Xinghao Zhu, Masayoshi Tomizuka and Diego Romeres
MERL Contacts: Radu Corcodel; Chiori Hori; Siddarth Jain; Devesh K. Jha; Diego Romeres
Research Areas: Artificial Intelligence, Machine Learning, RoboticsBrief- MERL Researchers received an "Honorable Mention award" at the Workshop on Instruction Tuning and Instruction Following at the NeurIPS 2023 conference in New Orleans. The workshop was on the topic of instruction tuning and Instruction following for Large Language Models (LLMs). MERL researchers presented their work on interactive planning using LLMs for partially observable robotic tasks during the oral presentation session at the workshop.
-
AWARD Joint University of Padua-MERL team wins Challenge 'AI Olympics With RealAIGym' Date: August 25, 2023
Awarded to: Alberto Dalla Libera, Niccolo' Turcato, Giulio Giacomuzzo, Ruggero Carli, Diego Romeres
MERL Contact: Diego Romeres
Research Areas: Artificial Intelligence, Machine Learning, RoboticsBrief- A joint team consisting of members of University of Padua and MERL ranked 1st in the IJCAI2023 Challenge "Al Olympics With RealAlGym: Is Al Ready for Athletic Intelligence in the Real World?". The team was composed by MERL researcher Diego Romeres and a team from University Padua (UniPD) consisting of Alberto Dalla Libera, Ph.D., Ph.D. Candidates: Niccolò Turcato, Giulio Giacomuzzo and Prof. Ruggero Carli from University of Padua.
The International Joint Conference on Artificial Intelligence (IJCAI) is a premier gathering for AI researchers and organizes several competitions. This year the competition CC7 "AI Olympics With RealAIGym: Is AI Ready for Athletic Intelligence in the Real World?" consisted of two stages: simulation and real-robot experiments on two under-actuated robotic systems. The two robotics systems were treated as separate tracks and one final winner was selected for each track based on specific performance criteria in the control tasks.
The UniPD-MERL team competed and won in both tracks. The team's system made strong use of a Model-based Reinforcement Learning algorithm called (MC-PILCO) that we recently published in the journal IEEE Transaction on Robotics.
- A joint team consisting of members of University of Padua and MERL ranked 1st in the IJCAI2023 Challenge "Al Olympics With RealAlGym: Is Al Ready for Athletic Intelligence in the Real World?". The team was composed by MERL researcher Diego Romeres and a team from University Padua (UniPD) consisting of Alberto Dalla Libera, Ph.D., Ph.D. Candidates: Niccolò Turcato, Giulio Giacomuzzo and Prof. Ruggero Carli from University of Padua.
-
AWARD MERL Researchers Win Best Workshop Poster Award at the 2023 IEEE International Conference on Robotics and Automation (ICRA) Date: June 2, 2023
Awarded to: Yuki Shirai, Devesh Jha, Arvind Raghunathan and Dennis Hong
MERL Contacts: Devesh K. Jha; Arvind Raghunathan
Research Areas: Artificial Intelligence, Optimization, RoboticsBrief- MERL's paper titled: "Closed-Loop Tactile Controller for Tool Manipulation" Won the Best Poster Award in the workshop on "Embracing contacts : Making robots physically interact with our world". First author and MERL intern, Yuki Shirai, was presented with the award at a ceremony held at ICRA in London. MERL researchers Devesh Jha, Principal Research Scientist, and Arvind Raghunathan, Senior Principal Research Scientist and Senior Team Leader as well as Prof. Dennis Hong of University of California, Los Angeles are also coauthors.
The paper presents a technique to manipulate an object using a tool in a closed-loop fashion using vision-based tactile sensors. More information about the workshop and the various speakers can be found here https://sites.google.com/view/icra2023embracingcontacts/home.
- MERL's paper titled: "Closed-Loop Tactile Controller for Tool Manipulation" Won the Best Poster Award in the workshop on "Embracing contacts : Making robots physically interact with our world". First author and MERL intern, Yuki Shirai, was presented with the award at a ceremony held at ICRA in London. MERL researchers Devesh Jha, Principal Research Scientist, and Arvind Raghunathan, Senior Principal Research Scientist and Senior Team Leader as well as Prof. Dennis Hong of University of California, Los Angeles are also coauthors.
See All Awards for Robotics -
-
News & Events
-
NEWS MERL at the International Conference on Robotics and Automation (ICRA) 2024 Date: May 13, 2024 - May 17, 2024
Where: Yokohama, Japan
MERL Contacts: Anoop Cherian; Radu Corcodel; Stefano Di Cairano; Chiori Hori; Siddarth Jain; Devesh K. Jha; Jonathan Le Roux; Diego Romeres; William S. Yerazunis
Research Areas: Artificial Intelligence, Machine Learning, Optimization, Robotics, Speech & AudioBrief- MERL made significant contributions to both the organization and the technical program of the International Conference on Robotics and Automation (ICRA) 2024, which was held in Yokohama, Japan from May 13th to May 17th.
MERL was a Bronze sponsor of the conference, and exhibited a live robotic demonstration, which attracted a large audience. The demonstration showcased an Autonomous Robotic Assembly technology executed on MELCO's Assista robot arm and was the collaborative effort of the Optimization and Robotics Team together with the Advanced Technology department at Mitsubishi Electric.
MERL researchers from the Optimization and Robotics, Speech & Audio, and Control for Autonomy teams also presented 8 papers and 2 invited talks covering topics on robotic assembly, applications of LLMs to robotics, human robot interaction, safe and robust path planning for autonomous drones, transfer learning, perception and tactile sensing.
- MERL made significant contributions to both the organization and the technical program of the International Conference on Robotics and Automation (ICRA) 2024, which was held in Yokohama, Japan from May 13th to May 17th.
-
NEWS Diego Romeres gave an invited talk at the Padua University's Seminar series on "AI in Action" Date: April 9, 2024
MERL Contact: Diego Romeres
Research Areas: Artificial Intelligence, Dynamical Systems, Machine Learning, Optimization, RoboticsBrief- Diego Romeres, Principal Research Scientist and Team Leader in the Optimization and Robotics Team, was invited to speak as a guest lecturer in the seminar series on "AI in Action" in the Department of Management and Engineering, at the University of Padua.
The talk, entitled "Machine Learning for Robotics and Automation" described MERL's recent research on machine learning and model-based reinforcement learning applied to robotics and automation.
- Diego Romeres, Principal Research Scientist and Team Leader in the Optimization and Robotics Team, was invited to speak as a guest lecturer in the seminar series on "AI in Action" in the Department of Management and Engineering, at the University of Padua.
See All News & Events for Robotics -
-
Internships
-
SA2181: Autonomous mobile robot data collection
MERL is seeking a highly motivated intern to collaborate in the collection of data for sensing, planning, and control methods in a robotic test-bed using Turtlebots at MERL. The ideal candidate is enrolled in a Masters/PhD program in Electrical, Mechanical, Aerospace Engineering, Robotics, Computer Science, or related program, with prior experience in motion planning, control, optimization, computer vision, and their application in mobile robots, including experimental validation. The candidate should be proficient in ROS, C/C++, and Python. The expected duration of the internship is 1-2 months, with a flexible start date in early summer to fall.
-
SA2073: Multimodal scene-understanding
We are looking for a graduate student interested in helping advance the field of multimodal scene understanding, with a focus on scene understanding using natural language for robot dialog and/or indoor monitoring using a large language model. The intern will collaborate with MERL researchers to derive and implement new models and optimization methods, conduct experiments, and prepare results for publication. Internships regularly lead to one or more publications in top-tier venues, which can later become part of the intern''s doctoral work. The ideal candidates are senior Ph.D. students with experience in deep learning for audio-visual, signal, and natural language processing. Good programming skills in Python and knowledge of deep learning frameworks such as PyTorch are essential. Multiple positions are available with flexible start date (not just Spring/Summer but throughout 2024) and duration (typically 3-6 months).
-
CA2132: Optimization Algorithms for Motion Planning and Predictive Control
MERL is looking for a highly motivated and qualified individual to work on tailored computational algorithms for optimization-based motion planning and predictive control applications in autonomous systems (vehicles, mobile robots). The ideal candidate should have experience in either one or multiple of the following topics: convex and non-convex optimization, stochastic predictive control (e.g., scenario trees), interaction-aware motion planning, machine learning, learning-based model predictive control, mathematical programs with complementarity constraints (MPCCs), optimal control, and real-time optimization. PhD students in engineering or mathematics, especially with a focus on research related to any of the above topics are encouraged to apply. Publication of relevant results in conference proceedings or journals is expected. Capability of implementing the designs and algorithms in MATLAB/Python is required; coding parts of the algorithms in C/C++ is a plus. The expected duration of the internship is 3 months, and the start date is flexible.
See All Internships for Robotics -
-
Recent Publications
- "Chance-Constrained Optimization for Contact-rich Systems using Mixed Integer Programming", Nonlinear Analysis: Hybrid Systems, December 2024.BibTeX TR2024-008 PDF
- @article{Shirai2024dec,
- author = {Shirai, Yuki and Jha, Devesh K. and Raghunathan, Arvind and Romeres, Diego},
- title = {Chance-Constrained Optimization for Contact-rich Systems using Mixed Integer Programming},
- journal = {Nonlinear Analysis: Hybrid Systems},
- year = 2024,
- month = dec,
- url = {https://www.merl.com/publications/TR2024-008}
- }
, - "A Robust Invariant Set Planner For Quadrotors", European Control Conference (ECC), June 2024.BibTeX TR2024-081 PDF
- @inproceedings{Greiff2024jun,
- author = {Greiff, Marcus and Weiss, Avishai and Berntorp, Karl and Di Cairano, Stefano}},
- title = {A Robust Invariant Set Planner For Quadrotors},
- booktitle = {European Control Conference (ECC)},
- year = 2024,
- month = jun,
- url = {https://www.merl.com/publications/TR2024-081}
- }
, - "Distributed Co-Design of Motors and Motions for Robotic Manipulators", European Control Conference (ECC), June 2024.BibTeX TR2024-083 PDF
- @inproceedings{Lu2024jun,
- author = {Lu, Zehui and Wang, Yebin and Sakamoto, Yusuke and Mou, Shaoshuai}},
- title = {Distributed Co-Design of Motors and Motions for Robotic Manipulators},
- booktitle = {European Control Conference (ECC)},
- year = 2024,
- month = jun,
- url = {https://www.merl.com/publications/TR2024-083}
- }
, - "Simultaneous State Estimation and Contact Detection for Legged Robots by Multiple-Model Kalman Filtering", European Control Conference (ECC), June 2024.BibTeX TR2024-085 PDF
- @inproceedings{Menner2024jun,
- author = {Menner, Marcel and Berntorp, Karl}},
- title = {Simultaneous State Estimation and Contact Detection for Legged Robots by Multiple-Model Kalman Filtering},
- booktitle = {European Control Conference (ECC)},
- year = 2024,
- month = jun,
- url = {https://www.merl.com/publications/TR2024-085}
- }
, - "Data-driven monitoring with mobile sensors and charging stations using multi-arm bandits and coordinated motion planners", American Control Conference (ACC), June 2024.BibTeX TR2024-078 PDF
- @inproceedings{Nayak2024jun,
- author = {Nayak, Siddharth and Greiff, Marcus and Raghunathan, Arvind and Di Cairano, Stefano and Vinod, Abraham P.}},
- title = {Data-driven monitoring with mobile sensors and charging stations using multi-arm bandits and coordinated motion planners},
- booktitle = {American Control Conference (ACC)},
- year = 2024,
- month = jun,
- url = {https://www.merl.com/publications/TR2024-078}
- }
, - "Graph-Based Analog Joint Source Channel Coding for 3D Haptic Communication", IEEE International Conference on Communications (ICC), June 2024.BibTeX TR2024-070 PDF
- @inproceedings{Fujihashi2024jun,
- author = {Fujihashi, Takuya and Koike-Akino, Toshiaki and Corcodel, Radu}},
- title = {Graph-Based Analog Joint Source Channel Coding for 3D Haptic Communication},
- booktitle = {IEEE International Conference on Communications (ICC)},
- year = 2024,
- month = jun,
- url = {https://www.merl.com/publications/TR2024-070}
- }
, - "Human Action Understanding-based Robot Planning using Multimodal LLM", IEEE International Conference on Robotics and Automation (ICRA), June 2024.BibTeX TR2024-066 PDF
- @inproceedings{Kambara2024jun,
- author = {Kambara, Motonari and Hori, Chiori and Sugiura, Komei and Ota, Kei and Jha, Devesh K. and Khurana, Sameer and Jain, Siddarth and Corcodel, Radu and Romeres, Diego and Le Roux, Jonathan}},
- title = {Human Action Understanding-based Robot Planning using Multimodal LLM},
- booktitle = {IEEE International Conference on Robotics and Automation (ICRA) Workshop},
- year = 2024,
- month = jun,
- url = {https://www.merl.com/publications/TR2024-066}
- }
, - "Robust In-Hand Manipulation with Extrinsic Contacts", IEEE International Conference on Robotics and Automation (ICRA), May 2024.BibTeX TR2024-061 PDF
- @inproceedings{Liang2024may,
- author = {Liang, Boyuan and Ota, Kei and Tomizuka, Masayoshi and Jha, Devesh K.}},
- title = {Robust In-Hand Manipulation with Extrinsic Contacts},
- booktitle = {IEEE International Conference on Robotics and Automation (ICRA)},
- year = 2024,
- month = may,
- url = {https://www.merl.com/publications/TR2024-061}
- }
,
- "Chance-Constrained Optimization for Contact-rich Systems using Mixed Integer Programming", Nonlinear Analysis: Hybrid Systems, December 2024.
-
Videos
-
Software & Data Downloads