Steered Diffusion

A Generalized Framework for Plug-and-Play Conditional Image Synthesis

MERL Researchers: Tim K. Marks, Anoop Cherian, Suhas Lohit, Ye Wang, Toshiaki Koike-Akino.
Joint work with: Nithin Gopalakrishnan Nair (Johns Hopkins University), Vishal M. Patel (Johns Hopkins University)

Search MERL publications by keyword: Computer Vision , Machine Learning , Artificial Intelligence , diffusion models , Generative AI , generative models

Capitalizing on the power of diffusion models that have been trained with unlabeled data for unconditional image generation, our work enables them to be repurposed for conditional image synthesis without the need for any retraining. Since no additional training is required for conditional generation, we call this zero-shot conditional image generation. Previous approaches utilizing diffusion models for zero-shot conditional generation can either perform either label-based generation or fine-grained conditional generation. In contrast to previous work, we propose a generic strategy that can synthesize images given any kind of conditions: label-based or text-based image generation, as well as fine-grained (image-to-image) generation and editing. As a result, our method can be used for a wide variety of photorealistic conditional image generation tasks, including image colorization, super-resolution, semantic generation, identity replication, and text-guided editing.

 

Steering diffusion models at inference time

The key idea in our method is to steer the image generation of the diffusion model at inference time using a pre-existing inverse model. We call our method plug-and-play because the pre-existing inverse model can be plugged in directly at inference time, with no need for any task-specific training. For complex tasks, the inverse model can be an off-the-shelf pretrained neural network; e.g., for semantic generation, the inverse model is a pre-existing semantic segmentation network. For simpler linear inverse problems, the inverse model is a simple linear model; e.g., for image colorization, the inverse model is a simple linear mapping from RGB to grayscale. In each case, our loss function utilizes the pre-existing inverse model that characterizes the conditional task. This loss modulates the sampling trajectory of the diffusion process.

 

During each step of the sampling process, we make a rough estimate of the denoised image using the diffusion model itself as a denoiser. We pass this denoised estimate through the pre-existing inverse model to estimate the label. We then backpropagate the error in the estimated label to find a direction of sampling, and we modulate the diffusion model's prediction at each timestep with this direction.

Results

We present results on a wide range of tasks including image inpainting, image colorization, image super-resolution, semantic generation, identity replication, and text-based image editing. Moreover, to evaluate the effectiveness of our method over other zero-shot techniques, we perform qualitative and quantitative evaluations for the tasks of super-resolution, colorization, and inpainting. Qualitative results are show above, and qualitative comparisons with previous diffusion-based plug-and-play models are shown below. Our results demonstrate clear qualitative and quantitative improvements over state-of-the-art diffusion-based plug-and-play models while adding negligible additional computational cost.

 



Software & Data Downloads



MERL News & Events

  •  NEWS    MERL researchers presenting four papers and organizing the VLAR-SMART101 Workshop at ICCV 2023
    Date: October 2, 2023 - October 6, 2023
    Where: Paris/France
    MERL Contacts: Moitreya Chatterjee; Anoop Cherian; Michael J. Jones; Toshiaki Koike-Akino; Suhas Lohit; Tim K. Marks; Pedro Miraldo; Kuan-Chuan Peng; Ye Wang
    Research Areas: Artificial Intelligence, Computer Vision, Machine Learning
    Brief
    • MERL researchers are presenting 4 papers and organizing the VLAR-SMART-101 workshop at the ICCV 2023 conference, which will be held in Paris, France October 2-6. ICCV is one of the most prestigious and competitive international conferences in computer vision. Details are provided below.

      1. Conference paper: “Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis,” by Nithin Gopalakrishnan Nair, Anoop Cherian, Suhas Lohit, Ye Wang, Toshiaki Koike-Akino, Vishal Patel, and Tim K. Marks

      Conditional generative models typically demand large annotated training sets to achieve high-quality synthesis. As a result, there has been significant interest in plug-and-play generation, i.e., using a pre-defined model to guide the generative process. In this paper, we introduce Steered Diffusion, a generalized framework for fine-grained photorealistic zero-shot conditional image generation using a diffusion model trained for unconditional generation. The key idea is to steer the image generation of the diffusion model during inference via designing a loss using a pre-trained inverse model that characterizes the conditional task. Our model shows clear qualitative and quantitative improvements over state-of-the-art diffusion-based plug-and-play models, while adding negligible computational cost.

      2. Conference paper: "BANSAC: A dynamic BAyesian Network for adaptive SAmple Consensus," by Valter Piedade and Pedro Miraldo

      We derive a dynamic Bayesian network that updates individual data points' inlier scores while iterating RANSAC. At each iteration, we apply weighted sampling using the updated scores. Our method works with or without prior data point scorings. In addition, we use the updated inlier/outlier scoring for deriving a new stopping criterion for the RANSAC loop. Our method outperforms the baselines in accuracy while needing less computational time.

      3. Conference paper: "Robust Frame-to-Frame Camera Rotation Estimation in Crowded Scenes," by Fabien Delattre, David Dirnfeld, Phat Nguyen, Stephen Scarano, Michael J. Jones, Pedro Miraldo, and Erik Learned-Miller

      We present a novel approach to estimating camera rotation in crowded, real-world scenes captured using a handheld monocular video camera. Our method uses a novel generalization of the Hough transform on SO3 to efficiently find the camera rotation most compatible with the optical flow. Because the setting is not addressed well by other data sets, we provide a new dataset and benchmark, with high-accuracy and rigorously annotated ground truth on 17 video sequences. Our method is more accurate by almost 40 percent than the next best method.

      4. Workshop paper: "Tensor Factorization for Leveraging Cross-Modal Knowledge in Data-Constrained Infrared Object Detection" by Manish Sharma*, Moitreya Chatterjee*, Kuan-Chuan Peng, Suhas Lohit, and Michael Jones

      While state-of-the-art object detection methods for RGB images have reached some level of maturity, the same is not true for Infrared (IR) images. The primary bottleneck towards bridging this gap is the lack of sufficient labeled training data in the IR images. Towards addressing this issue, we present TensorFact, a novel tensor decomposition method which splits the convolution kernels of a CNN into low-rank factor matrices with fewer parameters. This compressed network is first pre-trained on RGB images and then augmented with only a few parameters. This augmented network is then trained on IR images, while freezing the weights trained on RGB. This prevents it from over-fitting, allowing it to generalize better. Experiments show that our method outperforms state-of-the-art.

      5. “Vision-and-Language Algorithmic Reasoning (VLAR) Workshop and SMART-101 Challenge” by Anoop Cherian,  Kuan-Chuan Peng, Suhas Lohit, Tim K. Marks, Ram Ramrakhya, Honglu Zhou, Kevin A. Smith, Joanna Matthiesen, and Joshua B. Tenenbaum

      MERL researchers along with researchers from MIT, GeorgiaTech, Math Kangaroo USA, and Rutgers University are jointly organizing a workshop on vision-and-language algorithmic reasoning at ICCV 2023 and conducting a challenge based on the SMART-101 puzzles described in the paper: Are Deep Neural Networks SMARTer than Second Graders?. A focus of this workshop is to bring together outstanding faculty/researchers working at the intersections of vision, language, and cognition to provide their opinions on the recent breakthroughs in large language models and artificial general intelligence, as well as showcase their cutting edge research that could inspire the audience to search for the missing pieces in our quest towards solving the puzzle of artificial intelligence.

      Workshop link: https://wvlar.github.io/iccv23/
  •  


MERL Publications

  •  Nair, N.G., Cherian, A., Lohit, S., Wang, Y., Koike-Akino, T., Patel, V.M., Marks, T.K., "Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis", IEEE International Conference on Computer Vision (ICCV), October 2023, pp. 20850-20860.
    BibTeX TR2023-126 PDF Software Presentation
    • @inproceedings{Nair2023sep,
    • author = {Nair, Nithin Gopalakrishnan and Cherian, Anoop and Lohit, Suhas and Wang, Ye and Koike-Akino, Toshiaki and Patel, Vishal M. and Marks, Tim K.},
    • title = {Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesis},
    • booktitle = {Proceedings of the IEEE/CVF International Conference on Computer Vision},
    • year = 2023,
    • pages = {20850--20860},
    • month = oct,
    • publisher = {IEEE/CVF},
    • url = {https://www.merl.com/publications/TR2023-126}
    • }