NEWS    MERL researchers presenting four papers and organizing the VLAR-SMART101 Workshop at ICCV 2023

Date:

October 2, 2023 - October 6, 2023

Where:

Paris/France

Description:

MERL researchers are presenting 4 papers and organizing the VLAR-SMART-101 workshop at the ICCV 2023 conference, which will be held in Paris, France October 2-6. ICCV is one of the most prestigious and competitive international conferences in computer vision. Details are provided below.

1. Conference paper: “Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis,” by Nithin Gopalakrishnan Nair, Anoop Cherian, Suhas Lohit, Ye Wang, Toshiaki Koike-Akino, Vishal Patel, and Tim K. Marks

Conditional generative models typically demand large annotated training sets to achieve high-quality synthesis. As a result, there has been significant interest in plug-and-play generation, i.e., using a pre-defined model to guide the generative process. In this paper, we introduce Steered Diffusion, a generalized framework for fine-grained photorealistic zero-shot conditional image generation using a diffusion model trained for unconditional generation. The key idea is to steer the image generation of the diffusion model during inference via designing a loss using a pre-trained inverse model that characterizes the conditional task. Our model shows clear qualitative and quantitative improvements over state-of-the-art diffusion-based plug-and-play models, while adding negligible computational cost.

2. Conference paper: "BANSAC: A dynamic BAyesian Network for adaptive SAmple Consensus," by Valter Piedade and Pedro Miraldo

We derive a dynamic Bayesian network that updates individual data points' inlier scores while iterating RANSAC. At each iteration, we apply weighted sampling using the updated scores. Our method works with or without prior data point scorings. In addition, we use the updated inlier/outlier scoring for deriving a new stopping criterion for the RANSAC loop. Our method outperforms the baselines in accuracy while needing less computational time.

3. Conference paper: "Robust Frame-to-Frame Camera Rotation Estimation in Crowded Scenes," by Fabien Delattre, David Dirnfeld, Phat Nguyen, Stephen Scarano, Michael J. Jones, Pedro Miraldo, and Erik Learned-Miller

We present a novel approach to estimating camera rotation in crowded, real-world scenes captured using a handheld monocular video camera. Our method uses a novel generalization of the Hough transform on SO3 to efficiently find the camera rotation most compatible with the optical flow. Because the setting is not addressed well by other data sets, we provide a new dataset and benchmark, with high-accuracy and rigorously annotated ground truth on 17 video sequences. Our method is more accurate by almost 40 percent than the next best method.

4. Workshop paper: "Tensor Factorization for Leveraging Cross-Modal Knowledge in Data-Constrained Infrared Object Detection" by Manish Sharma*, Moitreya Chatterjee*, Kuan-Chuan Peng, Suhas Lohit, and Michael Jones

While state-of-the-art object detection methods for RGB images have reached some level of maturity, the same is not true for Infrared (IR) images. The primary bottleneck towards bridging this gap is the lack of sufficient labeled training data in the IR images. Towards addressing this issue, we present TensorFact, a novel tensor decomposition method which splits the convolution kernels of a CNN into low-rank factor matrices with fewer parameters. This compressed network is first pre-trained on RGB images and then augmented with only a few parameters. This augmented network is then trained on IR images, while freezing the weights trained on RGB. This prevents it from over-fitting, allowing it to generalize better. Experiments show that our method outperforms state-of-the-art.

5. “Vision-and-Language Algorithmic Reasoning (VLAR) Workshop and SMART-101 Challenge” by Anoop Cherian,  Kuan-Chuan Peng, Suhas Lohit, Tim K. Marks, Ram Ramrakhya, Honglu Zhou, Kevin A. Smith, Joanna Matthiesen, and Joshua B. Tenenbaum

MERL researchers along with researchers from MIT, GeorgiaTech, Math Kangaroo USA, and Rutgers University are jointly organizing a workshop on vision-and-language algorithmic reasoning at ICCV 2023 and conducting a challenge based on the SMART-101 puzzles described in the paper: Are Deep Neural Networks SMARTer than Second Graders?. A focus of this workshop is to bring together outstanding faculty/researchers working at the intersections of vision, language, and cognition to provide their opinions on the recent breakthroughs in large language models and artificial general intelligence, as well as showcase their cutting edge research that could inspire the audience to search for the missing pieces in our quest towards solving the puzzle of artificial intelligence.

Workshop link: https://wvlar.github.io/iccv23/

External Link:

https://iccv2023.thecvf.com/

Research Areas:

Artificial Intelligence, Computer Vision, Machine Learning

• Related Publications

   Nair, N.G., Cherian, A., Lohit, S., Wang, Y., Koike-Akino, T., Patel, V.M., Marks, T.K., "Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis", IEEE International Conference on Computer Vision (ICCV), October 2023, pp. 20850-20860.

  BibTeX TR2023-126 PDF Software Presentation

  • @inproceedings{Nair2023sep,
  • author = {Nair, Nithin Gopalakrishnan and Cherian, Anoop and Lohit, Suhas and Wang, Ye and Koike-Akino, Toshiaki and Patel, Vishal M. and Marks, Tim K.},
  • title = {Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis},
  • booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision},
  • year = 2023,
  • pages = {20850--20860},
  • month = oct,
  • publisher = {IEEE/CVF},
  • url = {https://www.merl.com/publications/TR2023-126}
  • }

   Sharma, M., Chatterjee, M., Peng, K.-C., Lohit, S., Jones, M.J., "Tensor Factorization for Leveraging Cross-Modal Knowledge in Data-Constrained Infrared Object Detection", IEEE International Conference on Computer Vision Workshops (ICCV), October 2023, pp. 924-932.

  BibTeX TR2023-125 PDF Presentation

  • @inproceedings{Sharma2023oct,
  • author = {Sharma, Manish and Chatterjee, Moitreya and Peng, Kuan-Chuan and Lohit, Suhas and Jones, Michael J.},
  • title = {Tensor Factorization for Leveraging Cross-Modal Knowledge in Data-Constrained Infrared Object Detection},
  • booktitle = {IEEE International Conference on Computer Vision Workshops (ICCV)},
  • year = 2023,
  • pages = {924--932},
  • month = oct,
  • url = {https://www.merl.com/publications/TR2023-125}
  • }

   Miraldo, P., Piedade, V., "BANSAC: A dynamic BAyesian Network for adaptive SAmple Consensus", IEEE International Conference on Computer Vision (ICCV), DOI: 10.1109/​ICCV51070.2023.00346, October 2023, pp. 3715-3724.

  BibTeX TR2023-124 PDF Video Software

  • @inproceedings{Miraldo2023oct,
  • author = {Miraldo, Pedro and Piedade, Valter},
  • title = {BANSAC: A dynamic BAyesian Network for adaptive SAmple Consensus},
  • booktitle = {IEEE International Conference on Computer Vision (ICCV)},
  • year = 2023,
  • pages = {3715--3724},
  • month = oct,
  • publisher = {IEEE/CVF},
  • doi = {10.1109/ICCV51070.2023.00346},
  • issn = {2380-7504},
  • isbn = {979-8-3503-0718-4},
  • url = {https://www.merl.com/publications/TR2023-124}
  • }

   Delattre, F., Dirnfeld, D., Nguyen, P., Scarano, S., Jones, M.J., Miraldo, P., Learned-Miller, E., "Robust Frame-to-Frame Camera Rotation Estimation in Crowded Scenes", IEEE International Conference on Computer Vision (ICCV), DOI: 10.1109/​ICCV51070.2023.00894, October 2023, pp. 3715-3724.

  BibTeX TR2023-123 PDF Video Software

  • @inproceedings{Delattre2023oct,
  • author = {Delattre, Fabien and Dirnfeld, David and Nguyen, Phat and Scarano, Stephen and Jones, Michael J. and Miraldo, Pedro and Learned-Miller, Erik},
  • title = {Robust Frame-to-Frame Camera Rotation Estimation in Crowded Scenes},
  • booktitle = {IEEE International Conference on Computer Vision (ICCV)},
  • year = 2023,
  • pages = {3715--3724},
  • month = oct,
  • publisher = {IEEE/CVF},
  • doi = {10.1109/ICCV51070.2023.00894},
  • issn = {2380-7504},
  • isbn = {979-8-3503-0718-4},
  • url = {https://www.merl.com/publications/TR2023-123}
  • }