Moitreya Chatterjee

Moitreya Chatterjee
Moitreya's Quick Links

  • Biography

    Moitreya's research interests are in computer vision, and multimodal machine learning with a particular emphasis on learning from audio-visual data. His PhD work received the Joan and Lalit Bahl Fellowship and the Thomas and Margaret Huang Research Award. Earlier, he earned a M.S. degree in Computer Science from the University of Southern California (USC), during which he received an Outstanding Paper Award from the ACM International Conference on Multimodal Interaction (ICMI).

  • Recent News & Events

    •  TALK    [MERL Seminar Series 2023] Dr. Tanmay Gupta presents talk titled Visual Programming - A compositional approach to building General Purpose Vision Systems
      Date & Time: Tuesday, October 31, 2023; 2:00 PM
      Speaker: Tanmay Gupta, Allen Institute for Artificial Intelligence
      MERL Host: Moitreya Chatterjee
      Research Areas: Artificial Intelligence, Computer Vision, Machine Learning
      Abstract
      • Building General Purpose Vision Systems (GPVs) that can perform a huge variety of tasks has been a long-standing goal for the computer vision community. However, end-to-end training of these systems to handle different modalities and tasks has proven to be extremely challenging. In this talk, I will describe a lucrative neuro-symbolic alternative to the common end-to-end learning paradigm called Visual Programming. Visual Programming is a general framework that leverages the code-generation abilities of LLMs, existing neural models, and non-differentiable programs to enable powerful applications. Some of these applications continue to remain elusive for the current generation of end-to-end trained GPVs.
    •  
    •  NEWS    MERL researchers presenting four papers and organizing the VLAR-SMART101 Workshop at ICCV 2023
      Date: October 2, 2023 - October 6, 2023
      Where: Paris/France
      MERL Contacts: Moitreya Chatterjee; Anoop Cherian; Michael J. Jones; Toshiaki Koike-Akino; Suhas Lohit; Tim K. Marks; Pedro Miraldo; Kuan-Chuan Peng; Ye Wang
      Research Areas: Artificial Intelligence, Computer Vision, Machine Learning
      Brief
      • MERL researchers are presenting 4 papers and organizing the VLAR-SMART-101 workshop at the ICCV 2023 conference, which will be held in Paris, France October 2-6. ICCV is one of the most prestigious and competitive international conferences in computer vision. Details are provided below.

        1. Conference paper: “Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis,” by Nithin Gopalakrishnan Nair, Anoop Cherian, Suhas Lohit, Ye Wang, Toshiaki Koike-Akino, Vishal Patel, and Tim K. Marks

        Conditional generative models typically demand large annotated training sets to achieve high-quality synthesis. As a result, there has been significant interest in plug-and-play generation, i.e., using a pre-defined model to guide the generative process. In this paper, we introduce Steered Diffusion, a generalized framework for fine-grained photorealistic zero-shot conditional image generation using a diffusion model trained for unconditional generation. The key idea is to steer the image generation of the diffusion model during inference via designing a loss using a pre-trained inverse model that characterizes the conditional task. Our model shows clear qualitative and quantitative improvements over state-of-the-art diffusion-based plug-and-play models, while adding negligible computational cost.

        2. Conference paper: "BANSAC: A dynamic BAyesian Network for adaptive SAmple Consensus," by Valter Piedade and Pedro Miraldo

        We derive a dynamic Bayesian network that updates individual data points' inlier scores while iterating RANSAC. At each iteration, we apply weighted sampling using the updated scores. Our method works with or without prior data point scorings. In addition, we use the updated inlier/outlier scoring for deriving a new stopping criterion for the RANSAC loop. Our method outperforms the baselines in accuracy while needing less computational time.

        3. Conference paper: "Robust Frame-to-Frame Camera Rotation Estimation in Crowded Scenes," by Fabien Delattre, David Dirnfeld, Phat Nguyen, Stephen Scarano, Michael J. Jones, Pedro Miraldo, and Erik Learned-Miller

        We present a novel approach to estimating camera rotation in crowded, real-world scenes captured using a handheld monocular video camera. Our method uses a novel generalization of the Hough transform on SO3 to efficiently find the camera rotation most compatible with the optical flow. Because the setting is not addressed well by other data sets, we provide a new dataset and benchmark, with high-accuracy and rigorously annotated ground truth on 17 video sequences. Our method is more accurate by almost 40 percent than the next best method.

        4. Workshop paper: "Tensor Factorization for Leveraging Cross-Modal Knowledge in Data-Constrained Infrared Object Detection" by Manish Sharma*, Moitreya Chatterjee*, Kuan-Chuan Peng, Suhas Lohit, and Michael Jones

        While state-of-the-art object detection methods for RGB images have reached some level of maturity, the same is not true for Infrared (IR) images. The primary bottleneck towards bridging this gap is the lack of sufficient labeled training data in the IR images. Towards addressing this issue, we present TensorFact, a novel tensor decomposition method which splits the convolution kernels of a CNN into low-rank factor matrices with fewer parameters. This compressed network is first pre-trained on RGB images and then augmented with only a few parameters. This augmented network is then trained on IR images, while freezing the weights trained on RGB. This prevents it from over-fitting, allowing it to generalize better. Experiments show that our method outperforms state-of-the-art.

        5. “Vision-and-Language Algorithmic Reasoning (VLAR) Workshop and SMART-101 Challenge” by Anoop Cherian,  Kuan-Chuan Peng, Suhas Lohit, Tim K. Marks, Ram Ramrakhya, Honglu Zhou, Kevin A. Smith, Joanna Matthiesen, and Joshua B. Tenenbaum

        MERL researchers along with researchers from MIT, GeorgiaTech, Math Kangaroo USA, and Rutgers University are jointly organizing a workshop on vision-and-language algorithmic reasoning at ICCV 2023 and conducting a challenge based on the SMART-101 puzzles described in the paper: Are Deep Neural Networks SMARTer than Second Graders?. A focus of this workshop is to bring together outstanding faculty/researchers working at the intersections of vision, language, and cognition to provide their opinions on the recent breakthroughs in large language models and artificial general intelligence, as well as showcase their cutting edge research that could inspire the audience to search for the missing pieces in our quest towards solving the puzzle of artificial intelligence.

        Workshop link: https://wvlar.github.io/iccv23/
    •  

    See All News & Events for Moitreya

  • MERL Publications

    •  Liu, X., Paul, S., Chatterjee, M., Cherian, A., "CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments", AAAI Conference on Artificial Intelligence, DOI: 10.1609/​aaai.v38i4.28167, December 2023, pp. 3765-3773.
      BibTeX TR2023-154 PDF
      • @inproceedings{Liu2023dec2,
      • author = {Liu, Xiulong and Paul, Sudipta and Chatterjee, Moitreya and Cherian, Anoop},
      • title = {CAVEN: An Embodied Conversational Agent for Efficient Audio-Visual Navigation in Noisy Environments},
      • booktitle = {Proceedings of the 38th AAAI Conference on Artificial Intelligence},
      • year = 2023,
      • pages = {3765--3773},
      • month = dec,
      • doi = {10.1609/aaai.v38i4.28167},
      • url = {https://www.merl.com/publications/TR2023-154}
      • }
    •  Sharma, M., Chatterjee, M., Peng, K.-C., Lohit, S., Jones, M.J., "Tensor Factorization for Leveraging Cross-Modal Knowledge in Data-Constrained Infrared Object Detection", IEEE International Conference on Computer Vision Workshops (ICCV), October 2023, pp. 924-932.
      BibTeX TR2023-125 PDF Presentation
      • @inproceedings{Sharma2023oct,
      • author = {Sharma, Manish and Chatterjee, Moitreya and Peng, Kuan-Chuan and Lohit, Suhas and Jones, Michael J.},
      • title = {Tensor Factorization for Leveraging Cross-Modal Knowledge in Data-Constrained Infrared Object Detection},
      • booktitle = {IEEE International Conference on Computer Vision Workshops (ICCV)},
      • year = 2023,
      • pages = {924--932},
      • month = oct,
      • url = {https://www.merl.com/publications/TR2023-125}
      • }
    •  Liu, X., Paul, S., Chatterjee, M., Cherian, A., "Active Sparse Conversations for Improved Audio-Visual Embodied Navigation", arXiv, June 2023.
      BibTeX arXiv
      • @inproceedings{Liu2023jun,
      • author = {Liu, Xiulong and Paul, Sudipta and Chatterjee, Moitreya and Cherian, Anoop},
      • title = {Active Sparse Conversations for Improved Audio-Visual Embodied Navigation},
      • booktitle = {arXiv},
      • year = 2023,
      • month = jun,
      • url = {https://arxiv.org/abs/2306.04047}
      • }
    •  Chatterjee, M., Ahuja, N., Cherian, A., "Learning Audio-Visual Dynamics Using Scene Graphs for Audio Source Separation", Advances in Neural Information Processing Systems (NeurIPS), November 2022.
      BibTeX TR2022-140 PDF Presentation
      • @inproceedings{Chatterjee2022nov,
      • author = {Chatterjee, Moitreya and Ahuja, Narendra and Cherian, Anoop},
      • title = {Learning Audio-Visual Dynamics Using Scene Graphs for Audio Source Separation},
      • booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
      • year = 2022,
      • month = nov,
      • url = {https://www.merl.com/publications/TR2022-140}
      • }
    •  Chatterjee, M., Ahuja, N., Cherian, A., "Quantifying Predictive Uncertainty for Stochastic Video Synthesis from Audio", IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), June 2022.
      BibTeX TR2022-082 PDF
      • @inproceedings{Chatterjee2022jun,
      • author = {Chatterjee, Moitreya and Ahuja, Narendra and Cherian, Anoop},
      • title = {Quantifying Predictive Uncertainty for Stochastic Video Synthesis from Audio},
      • booktitle = {IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)},
      • year = 2022,
      • month = jun,
      • url = {https://www.merl.com/publications/TR2022-082}
      • }
    See All MERL Publications for Moitreya

  • Other Publications

    •  Moitreya Chatterjee, Narendra Ahuja and Anoop Cherian, "A hierarchical variational neural uncertainty model for stochastic video prediction", Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9751-9761.
      BibTeX
      • @Inproceedings{chatterjee2021hierarchical,
      • author = {Chatterjee, Moitreya and Ahuja, Narendra and Cherian, Anoop},
      • title = {A hierarchical variational neural uncertainty model for stochastic video prediction},
      • booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision},
      • year = 2021,
      • pages = {9751--9761}
      • }
    •  Moitreya Chatterjee, Jonathan Le Roux, Narendra Ahuja and Anoop Cherian, "Visual scene graphs for audio source separation", Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 1204-1213.
      BibTeX
      • @Inproceedings{chatterjee2021visual,
      • author = {Chatterjee, Moitreya and Le Roux, Jonathan and Ahuja, Narendra and Cherian, Anoop},
      • title = {Visual scene graphs for audio source separation},
      • booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision},
      • year = 2021,
      • pages = {1204--1213}
      • }
    •  Shijie Geng, Peng Gao, Moitreya Chatterjee, Chiori Hori, Jonathan Le Roux, Yongfeng Zhang, Hongsheng Li and Anoop Cherian, "Dynamic graph representation learning for video dialog via multi-modal shuffled transformers", Proceedings of the AAAI Conference on Artificial Intelligence, 2021, vol. 35, pp. 1415-1423.
      BibTeX
      • @Inproceedings{geng2021dynamic,
      • author = {Geng, Shijie and Gao, Peng and Chatterjee, Moitreya and Hori, Chiori and Le Roux, Jonathan and Zhang, Yongfeng and Li, Hongsheng and Cherian, Anoop},
      • title = {Dynamic graph representation learning for video dialog via multi-modal shuffled transformers},
      • booktitle = {Proceedings of the AAAI Conference on Artificial Intelligence},
      • year = 2021,
      • volume = 35,
      • number = 2,
      • pages = {1415--1423}
      • }
    •  Moitreya Chatterjee and Anoop Cherian, "Sound2sight: Generating visual dynamics from sound and context", European Conference on Computer Vision, 2020, pp. 701-719.
      BibTeX
      • @Inproceedings{chatterjee2020sound2sight,
      • author = {Chatterjee, Moitreya and Cherian, Anoop},
      • title = {Sound2sight: Generating visual dynamics from sound and context},
      • booktitle = {European Conference on Computer Vision},
      • year = 2020,
      • pages = {701--719},
      • organization = {Springer}
      • }
    •  Abhimanyu Dubey, Moitreya Chatterjee and Narendra Ahuja, "Coreset-based neural network compression", Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 454-470.
      BibTeX
      • @Inproceedings{dubey2018coreset,
      • author = {Dubey, Abhimanyu and Chatterjee, Moitreya and Ahuja, Narendra},
      • title = {Coreset-based neural network compression},
      • booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
      • year = 2018,
      • pages = {454--470}
      • }
    •  Arulkumar Subramaniam, Moitreya Chatterjee and Anurag Mittal, "Deep neural networks with inexact matching for person re-identification", Advances in neural information processing systems, Vol. 29, 2016.
      BibTeX
      • @Article{subramaniam2016deep,
      • author = {Subramaniam, Arulkumar and Chatterjee, Moitreya and Mittal, Anurag},
      • title = {Deep neural networks with inexact matching for person re-identification},
      • journal = {Advances in neural information processing systems},
      • year = 2016,
      • volume = 29
      • }
    •  Moitreya Chatterjee, Sunghyun Park, Louis-Philippe Morency and Stefan Scherer, "Combining two perspectives on classifying multimodal data for recognizing speaker traits", Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, 2015, pp. 7-14.
      BibTeX
      • @Inproceedings{chatterjee2015combining,
      • author = {Chatterjee, Moitreya and Park, Sunghyun and Morency, Louis-Philippe and Scherer, Stefan},
      • title = {Combining two perspectives on classifying multimodal data for recognizing speaker traits},
      • booktitle = {Proceedings of the 2015 ACM on International Conference on Multimodal Interaction},
      • year = 2015,
      • pages = {7--14}
      • }

  • Videos