Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion

Jixuan He*1,2, Wanhua Li*1, Ye Liu1,3, Junsik Kim1, Donglai Wei4, Hanspeter Pfister1,
1Harvard University, 2Cornell Tech, 3The Hong Kong Polytechnic University, 4Boston College, *Equal Contribution
arXiv Demo Code Data

Madd performs affordance-aware object insertion conditioning on different position prompts, including points, bounding boxes, masks, and even null prompts.

Abstract

As a common image editing operation, image composition involves integrating foreground objects into background scenes. In this paper, we expand the application of the concept of Affordance from human-centered image composition tasks to a more general object-scene composition framework, addressing the complex interplay between foreground objects and background scenes.

Following the principle of Affordance, we define the affordance-aware object insertion task, which aims to seamlessly insert any object into any scene with various position prompts. To address the limited data issue and incorporate this task, we constructed the SAM-FB dataset, which contains over 3 million examples across more than 3,000 object categories. Furthermore, we propose the Mask-Aware Dual Diffusion (Madd) model, which utilizes a dual-stream architecture to simultaneously denoise the RGB image and the insertion mask.

By explicitly modeling the insertion mask in the diffusion process, Madd effectively facilitates the notion of affordance. Extensive experimental results show that our method outperforms the state-of-the-art methods and exhibits strong generalization performance on in-the-wild images.

Affordance-aware Insertion

Model Architecture

The framework of Madd. Foreground objects are encoded using a DINOv2 encoder, serving as the guidance signal through the cross-attention mechanism. The position prompt encoder unifies different types of position prompts, which are then concatenated with the latent mask mt. The background is encoded using a VAE encoder and then concatenated with the latent image zt. We use a dual branch structure to denoise RGB image z and object mask m simultaneously.

More Visualization

SAM-FB Test Set

We presents the visualization results on the SAM-FB test set. In each group, the leftmost image depicts the back- ground marked with a position prompt. Our Madd predicts the RGB image and mask of the inserted object, which are shown in the last two images of each group.


In-the-wild Images

Example of in-the-wild insertion results with details. Madd could keep the appearance better and adjust the foreground's properties better compared with different baseline models on both common and uncommon objects. In the last row, Madd generated reasonable insertion when provided ambiguous prompts.

BibTeX

      
        @article{he2024affordanceawareobjectinsertionmaskaware,
          author    = {Jixuan He and Wanhua Li and Ye Liu and Junsik Kim and Donglai Wei and Hanspeter Pfister},
          title     = {Affordance-Aware Object Insertion via Mask-Aware Dual Diffusion},
          journal   = {arXiv preprint arXiv:2412.14462},
          year      = {2024},
        }