Computational Sensing

Utilizing computation to improve sensing capabilities.

Our research in the area of computational sensing focuses on signal acquisition and design, signal modeling and reconstruction algorithms, including inverse problems, as well as array signal processing techniques.

  • Quick Links

  • Researchers

  • Awards

    •  AWARD    MERL’s Paper on Wi-Fi Sensing Earns Top 3% Paper Recognition at ICASSP 2023, Selected as a Best Student Paper Award Finalist
      Date: June 9, 2023
      Awarded to: Cristian J. Vaca-Rubio, Pu Wang, Toshiaki Koike-Akino, Ye Wang, Petros Boufounos and Petar Popovski
      MERL Contacts: Petros T. Boufounos; Toshiaki Koike-Akino; Pu (Perry) Wang; Ye Wang
      Research Areas: Artificial Intelligence, Communications, Computational Sensing, Dynamical Systems, Machine Learning, Signal Processing
      Brief
      • A MERL Paper on Wi-Fi sensing was recognized as a Top 3% Paper among all 2709 accepted papers at the 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2023). Co-authored by Cristian Vaca-Rubio and Petar Popovski from Aalborg University, Denmark, and MERL researchers Pu Wang, Toshiaki Koike-Akino, Ye Wang, and Petros Boufounos, the paper "MmWave Wi-Fi Trajectory Estimation with Continous-Time Neural Dynamic Learning" was also a Best Student Paper Award finalist.

        Performed during Cristian’s stay at MERL first as a visiting Marie Skłodowska-Curie Fellow and then as a full-time intern in 2022, this work capitalizes on standards-compliant Wi-Fi signals to perform indoor localization and sensing. The paper uses a neural dynamic learning framework to address technical issues such as low sampling rate and irregular sampling intervals.

        ICASSP, a flagship conference of the IEEE Signal Processing Society (SPS), was hosted on the Greek island of Rhodes from June 04 to June 10, 2023. ICASSP 2023 marked the largest ICASSP in history, boasting over 4000 participants and 6128 submitted papers, out of which 2709 were accepted.
    •  
    •  AWARD    Joshua Rapp wins Best Dissertation Award from the IEEE Signal Processing Society
      Date: December 20, 2021
      Awarded to: Joshua Rapp
      MERL Contact: Joshua Rapp
      Research Areas: Computational Sensing, Signal Processing
      Brief
      • Joshua Rapp has won the 2021 Best PhD Dissertation Award from the IEEE Signal Processing Society.
        The award recognizes a PhD thesis completed on a signal processing subject within the past three years for its relevant work in signal processing while stimulating further research in the field.

        Dr. Rapp completed his PhD at Boston University in 2020 with a thesis entitled "Probabilistic Modeling for Single-Photon Lidar." The dissertation tackles challenges of the acquisition and processing of 3D depth maps reconstructed from time-of-flight data captured one photon at a time.
        The award will be presented at the 2022 IEEE International Conference on Image Processing (ICIP) in France.
    •  
    •  AWARD    Petros Boufounos Elevated to IEEE Fellow
      Date: January 1, 2022
      Awarded to: Petros T. Boufounos
      MERL Contact: Petros T. Boufounos
      Research Areas: Computational Sensing, Signal Processing
      Brief
      • MERL’s Petros Boufounos has been elevated to IEEE Fellow, effective January 2022, for “contributions to compressed sensing.”

        IEEE Fellow is the highest grade of membership of the IEEE. It honors members with an outstanding record of technical achievements, contributing importantly to the advancement or application of engineering, science and technology, and bringing significant value to society. Each year, following a rigorous evaluation procedure, the IEEE Fellow Committee recommends a select group of recipients for elevation to IEEE Fellow. Less than 0.1% of voting members are selected annually for this member grade elevation.
    •  

    See All Awards for Computational Sensing

  • News & Events

    •  NEWS    MERL Papers and Workshops at CVPR 2024
      Date: June 17, 2024 - June 21, 2024
      Where: Seattle, WA
      MERL Contacts: Petros T. Boufounos; Moitreya Chatterjee; Anoop Cherian; Michael J. Jones; Toshiaki Koike-Akino; Jonathan Le Roux; Suhas Lohit; Tim K. Marks; Pedro Miraldo; Jing Liu; Kuan-Chuan Peng; Pu (Perry) Wang; Ye Wang; Matthew Brand
      Research Areas: Artificial Intelligence, Computational Sensing, Computer Vision, Machine Learning, Speech & Audio
      Brief
      • MERL researchers are presenting 5 conference papers, 3 workshop papers, and are co-organizing two workshops at the CVPR 2024 conference, which will be held in Seattle, June 17-21. CVPR is one of the most prestigious and competitive international conferences in computer vision. Details of MERL contributions are provided below.

        CVPR Conference Papers:

        1. "TI2V-Zero: Zero-Shot Image Conditioning for Text-to-Video Diffusion Models" by H. Ni, B. Egger, S. Lohit, A. Cherian, Y. Wang, T. Koike-Akino, S. X. Huang, and T. K. Marks

        This work enables a pretrained text-to-video (T2V) diffusion model to be additionally conditioned on an input image (first video frame), yielding a text+image to video (TI2V) model. Other than using the pretrained T2V model, our method requires no ("zero") training or fine-tuning. The paper uses a "repeat-and-slide" method and diffusion resampling to synthesize videos from a given starting image and text describing the video content.

        Paper: https://www.merl.com/publications/TR2024-059
        Project page: https://merl.com/research/highlights/TI2V-Zero

        2. "Long-Tailed Anomaly Detection with Learnable Class Names" by C.-H. Ho, K.-C. Peng, and N. Vasconcelos

        This work aims to identify defects across various classes without relying on hard-coded class names. We introduce the concept of long-tailed anomaly detection, addressing challenges like class imbalance and dataset variability. Our proposed method combines reconstruction and semantic modules, learning pseudo-class names and utilizing a variational autoencoder for feature synthesis to improve performance in long-tailed datasets, outperforming existing methods in experiments.

        Paper: https://www.merl.com/publications/TR2024-040

        3. "Gear-NeRF: Free-Viewpoint Rendering and Tracking with Motion-aware Spatio-Temporal Sampling" by X. Liu, Y-W. Tai, C-T. Tang, P. Miraldo, S. Lohit, and M. Chatterjee

        This work presents a new strategy for rendering dynamic scenes from novel viewpoints. Our approach is based on stratifying the scene into regions based on the extent of motion of the region, which is automatically determined. Regions with higher motion are permitted a denser spatio-temporal sampling strategy for more faithful rendering of the scene. Additionally, to the best of our knowledge, ours is the first work to enable tracking of objects in the scene from novel views - based on the preferences of a user, provided by a click.

        Paper: https://www.merl.com/publications/TR2024-042

        4. "SIRA: Scalable Inter-frame Relation and Association for Radar Perception" by R. Yataka, P. Wang, P. T. Boufounos, and R. Takahashi

        Overcoming the limitations on radar feature extraction such as low spatial resolution, multipath reflection, and motion blurs, this paper proposes SIRA (Scalable Inter-frame Relation and Association) for scalable radar perception with two designs: 1) extended temporal relation, generalizing the existing temporal relation layer from two frames to multiple inter-frames with temporally regrouped window attention for scalability; and 2) motion consistency track with a pseudo-tracklet generated from observational data for better object association.

        Paper: https://www.merl.com/publications/TR2024-041

        5. "RILA: Reflective and Imaginative Language Agent for Zero-Shot Semantic Audio-Visual Navigation" by Z. Yang, J. Liu, P. Chen, A. Cherian, T. K. Marks, J. L. Roux, and C. Gan

        We leverage Large Language Models (LLM) for zero-shot semantic audio visual navigation. Specifically, by employing multi-modal models to process sensory data, we instruct an LLM-based planner to actively explore the environment by adaptively evaluating and dismissing inaccurate perceptual descriptions.

        Paper: https://www.merl.com/publications/TR2024-043

        CVPR Workshop Papers:

        1. "CoLa-SDF: Controllable Latent StyleSDF for Disentangled 3D Face Generation" by R. Dey, B. Egger, V. Boddeti, Y. Wang, and T. K. Marks

        This paper proposes a new method for generating 3D faces and rendering them to images by combining the controllability of nonlinear 3DMMs with the high fidelity of implicit 3D GANs. Inspired by StyleSDF, our model uses a similar architecture but enforces the latent space to match the interpretable and physical parameters of the nonlinear 3D morphable model MOST-GAN.

        Paper: https://www.merl.com/publications/TR2024-045

        2. “Tracklet-based Explainable Video Anomaly Localization” by A. Singh, M. J. Jones, and E. Learned-Miller

        This paper describes a new method for localizing anomalous activity in video of a scene given sample videos of normal activity from the same scene. The method is based on detecting and tracking objects in the scene and estimating high-level attributes of the objects such as their location, size, short-term trajectory and object class. These high-level attributes can then be used to detect unusual activity as well as to provide a human-understandable explanation for what is unusual about the activity.

        Paper: https://www.merl.com/publications/TR2024-057

        MERL co-organized workshops:

        1. "Multimodal Algorithmic Reasoning Workshop" by A. Cherian, K-C. Peng, S. Lohit, M. Chatterjee, H. Zhou, K. Smith, T. K. Marks, J. Mathissen, and J. Tenenbaum

        Workshop link: https://marworkshop.github.io/cvpr24/index.html

        2. "The 5th Workshop on Fair, Data-Efficient, and Trusted Computer Vision" by K-C. Peng, et al.

        Workshop link: https://fadetrcv.github.io/2024/

        3. "SuperLoRA: Parameter-Efficient Unified Adaptation for Large Vision Models" by X. Chen, J. Liu, Y. Wang, P. Wang, M. Brand, G. Wang, and T. Koike-Akino

        This paper proposes a generalized framework called SuperLoRA that unifies and extends different variants of low-rank adaptation (LoRA). Introducing new options with grouping, folding, shuffling, projection, and tensor decomposition, SuperLoRA offers high flexibility and demonstrates superior performance up to 10-fold gain in parameter efficiency for transfer learning tasks.

        Paper: https://www.merl.com/publications/TR2024-062
    •  
    •  TALK    [MERL Seminar Series 2024] Fadel Adib presents talk titled Decoding Hidden Worlds: Unprecedented Sensing and Connectivity for Climate, Robotics, & Smart Environments
      Date & Time: Wednesday, April 3, 2024; 12:00 PM
      Speaker: Fadel Adib, MIT & Cartesian
      MERL Host: Wael Hajj Ali
      Research Areas: Computational Sensing, Dynamical Systems, Signal Processing
      Abstract
      • This talk will cover a new generation of technologies that can sense, connect, and perceive the physical world in unprecedented ways. These technologies can uncover hidden worlds around us, promising transformative impact on areas spanning climate change monitoring, ocean mapping, healthcare, food security, supply chain, and even extraterrestrial exploration.

        The talk will cover four core technologies invented by Prof. Adib and his team. The first is an ocean internet-of-things (IoT) that uses battery-free sensors for climate change monitoring, marine life discovery, and seafood production (aquaculture). The second is a new perception technology that enables robots to sense and manipulate hidden objects. The third is a new augmented reality headset with ``X-ray vision”, which extends human perception beyond line-of-sight. The fourth is a wireless sensing technology that can “see through walls” and monitor people’s vital signs (including their breathing, heart rate, and emotions), enabling smart environments that sense humans requiring any contact with the human body.

        The talk will touch on the journey of these technologies from their inception at MIT to international collaborations and startups that are translating them to real-world impact in areas spanning healthcare, climate change, and supply chain.
    •  

    See All News & Events for Computational Sensing

  • Research Highlights

  • Internships

    • ST2090: Radiation Source Localization

      The Computational Sensing Team at MERL is seeking an intern to work on estimation algorithms for radioactive source localization. The candidate should have experience with statistical modeling and estimation theory. A detailed knowledge of interactions of particles with matter, imaging inverse problems, and/or computed tomography is preferred. Hands-on experience with high-energy physics simulators (e.g., Geant4) is beneficial but not required. Strong programming skills in Python are essential. Publication of the results produced during our internships is expected. The duration is anticipated to be 3-6 months.

    • ST2083: Deep Learning for Radar Perception

      The Computation Sensing team at MERL is seeking a highly motivated intern to conduct fundamental research in radar perception. Expertise in deep learning-based object detection, multiple object tracking, data association, and representation learning (detection points, heatmaps, and raw radar waveforms) is required. Previous hands-on experience on open indoor/outdoor radar datasets is a plus. Familiarity with the concept of FMCW, MIMO, and range-Doppler-angle spectrum is an asset. The intern will collaborate with a small group of MERL researchers to develop novel algorithms, design experiments with MERL in-house testbed, and prepare results for patents and publication. The expected duration of the internship is 3 months with a flexible start date.

    • ST1762: Computational Sensing Technologies

      The Computational Sensing team at MERL is seeking motivated and qualified individuals to assist in the development of computational methods for a variety of sensing applications. Ideal candidates should be Ph.D. students and have solid background and publication record in any of the following, or related areas: imaging inverse problems, deep learning for inverse problems, large-scale optimization, blind inverse scattering, radar/lidar/THz imaging, joint communications and sensing, multimodal sensor fusion, object or human tracking, sensing of dynamical systems, or wave-based inversion. Experience with experimentally measured data is desirable. Publication of the results produced during our internships is expected. The duration of the internships is anticipated to be 3-6 months. Start date is flexible.


    See All Internships for Computational Sensing

  • Recent Publications

    •  Yataka, R., Wang, P., Boufounos, P.T., Takahashi, R., "SIRA: Scalable Inter-frame Relation and Association for Radar Perception", IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2024.
      BibTeX TR2024-041 PDF
      • @inproceedings{Yataka2024jun,
      • author = {Yataka, Ryoma and Wang, Pu and Boufounos, Petros T. and Takahashi, Ryuhei},
      • title = {SIRA: Scalable Inter-frame Relation and Association for Radar Perception},
      • booktitle = {IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
      • year = 2024,
      • month = jun,
      • url = {https://www.merl.com/publications/TR2024-041}
      • }
    •  Vaca-Rubio, C., Wang, P., Koike-Akino, T., Wang, Y., Boufounos, P.T., Popovski, P., "Object Trajectory Estimation with Continuous-Time Neural Dynamic Learning of Millimeter-Wave Wi-Fi", IEEE Journal of Selected Topics in Signal Processing, April 2024.
      BibTeX TR2024-044 PDF
      • @article{Vaca-Rubio2024apr,
      • author = {Vaca-Rubio, Cristian and Wang, Pu and Koike-Akino, Toshiaki and Wang, Ye and Boufounos, Petros T. and Popovski, Petar},
      • title = {Object Trajectory Estimation with Continuous-Time Neural Dynamic Learning of Millimeter-Wave Wi-Fi},
      • journal = {IEEE Journal of Selected Topics in Signal Processing},
      • year = 2024,
      • month = apr,
      • url = {https://www.merl.com/publications/TR2024-044}
      • }
    •  Fernandez-Menduina, S., Rapp, J., Mansour, H., Greiff, M., Parsons, K., "Tracking Beyond the Unambiguous Range with Modulo Single-Photon Lidar", IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), DOI: 10.1109/​ICASSP48485.2024.10446835, March 2024, pp. 6-10.
      BibTeX TR2024-021 PDF
      • @inproceedings{Fernandez-Menduina2024mar,
      • author = {Fernandez-Menduina, Samuel and Rapp, Joshua and Mansour, Hassan and Greiff, Marcus and Parsons, Kieran},
      • title = {Tracking Beyond the Unambiguous Range with Modulo Single-Photon Lidar},
      • booktitle = {IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)},
      • year = 2024,
      • pages = {6--10},
      • month = mar,
      • doi = {10.1109/ICASSP48485.2024.10446835},
      • url = {https://www.merl.com/publications/TR2024-021}
      • }
    •  Sholokhov, A., Rapp, J., Nabi, S., Brunton, S., Kutz, N., Mansour, H., "Single-pixel imaging of dynamic flows using Neural ODE regularization", IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), DOI: 10.1109/​ICASSP48485.2024.10447584, March 2024, pp. 2530-2534.
      BibTeX TR2024-024 PDF
      • @inproceedings{Sholokhov2024mar,
      • author = {Sholokhov, Aleksei and Rapp, Joshua and Nabi, Saleh and Brunton, Steven and Kutz, Nathan and Mansour, Hassan},
      • title = {Single-pixel imaging of dynamic flows using Neural ODE regularization},
      • booktitle = {IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)},
      • year = 2024,
      • pages = {2530--2534},
      • month = mar,
      • publisher = {IEEE},
      • doi = {10.1109/ICASSP48485.2024.10447584},
      • url = {https://www.merl.com/publications/TR2024-024}
      • }
    •  Kato, S., Wang, P., Koike-Akino, T., Fujihashi, T., Mansour, H., Boufounos, P.T., "Object Trajectory Estimation with Multi-Band Wi-Fi Neural Dynamic Fusion", IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), DOI: 10.1109/​ICASSP48485.2024.10445972, March 2024, pp. 13261-13265.
      BibTeX TR2024-019 PDF
      • @inproceedings{Kato2024mar,
      • author = {Kato, Sorachi and Wang, Pu and Koike-Akino, Toshiaki and Fujihashi, Takuya and Mansour, Hassan and Boufounos, Petros T.},
      • title = {Object Trajectory Estimation with Multi-Band Wi-Fi Neural Dynamic Fusion},
      • booktitle = {IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)},
      • year = 2024,
      • pages = {13261--13265},
      • month = mar,
      • publisher = {IEEE},
      • doi = {10.1109/ICASSP48485.2024.10445972},
      • issn = {2379-190X},
      • isbn = {979-8-3503-4485-1},
      • url = {https://www.merl.com/publications/TR2024-019}
      • }
    •  Wang, P., Boufounos, P.T., "Monostatic DMG Passive Sensing with Hypothesis Testing", IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), DOI: 10.1109/​ICASSP48485.2024.10447134, March 2024, pp. 13381-13385.
      BibTeX TR2024-020 PDF
      • @inproceedings{Wang2024mar,
      • author = {Wang, Pu and Boufounos, Petros T.},
      • title = {Monostatic DMG Passive Sensing with Hypothesis Testing},
      • booktitle = {IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)},
      • year = 2024,
      • pages = {13381--13385},
      • month = mar,
      • publisher = {IEEE},
      • doi = {10.1109/ICASSP48485.2024.10447134},
      • issn = {2379-190X},
      • isbn = {979-8-3503-4485-1},
      • url = {https://www.merl.com/publications/TR2024-020}
      • }
    •  Yataka, R., Wang, P., Boufounos, P.T., Takahashi, R., "Radar Perception with Scalable Connective Temporal Relations for Autonomous Driving", IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), DOI: 10.1109/​ICASSP48485.2024.10446449, March 2024, pp. 13266-13270.
      BibTeX TR2024-023 PDF
      • @inproceedings{Yataka2024mar,
      • author = {Yataka, Ryoma and Wang, Pu and Boufounos, Petros T. and Takahashi, Ryuhei},
      • title = {Radar Perception with Scalable Connective Temporal Relations for Autonomous Driving},
      • booktitle = {IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP)},
      • year = 2024,
      • pages = {13266--13270},
      • month = mar,
      • publisher = {IEEE},
      • doi = {10.1109/ICASSP48485.2024.10446449},
      • issn = {2379-190X},
      • isbn = {979-8-3503-4485-1},
      • url = {https://www.merl.com/publications/TR2024-023}
      • }
    •  Ma, Y., Rapp, J., Boufounos, P.T., Mansour, H., "A model of spatial resolution uncertainty for Compton camera imaging", International Conference on Advancements in Nuclear Instrumentation Measurement Methods and their Applications (ANIMMA), DOI: 10.1051/​epjconf/​202328810002, January 2024, pp. 10002.
      BibTeX TR2024-005 PDF
      • @inproceedings{Ma2024jan,
      • author = {Ma, Yanting and Rapp, Joshua and Boufounos, Petros T. and Mansour, Hassan},
      • title = {A model of spatial resolution uncertainty for Compton camera imaging},
      • booktitle = {Advancements in Nuclear Instrumentation Measurement Methods and their Applications (ANIMMA)},
      • year = 2024,
      • pages = 10002,
      • month = jan,
      • publisher = {EPJ Web of Conferences, 288},
      • doi = {10.1051/epjconf/202328810002},
      • url = {https://www.merl.com/publications/TR2024-005}
      • }
    See All Publications for Computational Sensing

  • Videos

  • Software & Data Downloads