TurboEdit: Instant text-based image editing

Zongze Wu, Nicholas Kolkin, Jonathan Brandt, Richard Zhang, Eli Shechtman
Adobe Research
To appear in ECCV, 2024
arXiv

Abstract

We address the challenges of precise image inversion and disentangled image editing in the context of few-step diffusion models. We introduce an encoder based iterative inversion technique. The inversion network is conditioned on the input image and the reconstructed image from the previous step, allowing for correction of the next reconstruction towards the input image. We demonstrate that disentangled controls can be easily achieved in the few-step diffusion model by conditioning on an (automatically generated) detailed text prompt. To manipulate the inverted image, we freeze the noise maps and modify one attribute in the text prompt (either manually or via instruction based editing driven by an LLM), resulting in the generation of a new image similar to the input image with only one attribute changed. It can further control the editing strength and accept instructive text prompt. Our approach facilitates realistic text-guided image edits in real-time, requiring only 8 number of functional evaluations (NFEs) in inversion (one-time cost) and 4 NFEs per edit. Our method is not only fast, but also significantly outperforms state-of-the-art multi-step diffusion editing techniques.

Method

Given an input real image x0, we utilize the LLaVA to generate a detailed caption c. Users can modify c to create a new text prompt c ′. The inversion process begins by feeding the x0, c, current time step t, and a previously reconstructed image x0,t+1 (initialized as a zero matrix) into the inversion network. This network then predicts the noise ϵt, which is subsequently input into a frozen SDXL-Turbo model to generate the new reconstruction image x0,t. Given the final inverted noise ϵt, along with c, we can use SDXL-Turbo to create an inversion trajectory and reconstruct x0,0, which is very similar to x0. Using the same noises ϵt and slightly different text prompt c ′, starting from t = T to smaller t, the editing trajectory will be very similar to the inversion trajectory, and the generated image will closely resemble the input image, differing only in the specified attribute in c'.

BibTeX

@article{wu2024turboedit,
  author    = {Wu, Zongze and Kolkin, Nicholas and Brandt, Jonathan and Zhang, Richard and Shechtman, Eli},
  title     = {TurboEdit: Instant text-based image editing},
  journal   = {ECCV},
  year      = {2024},
}

Related Links

Few step diffusion model SDXL-Turbo.

StyleGAN based iterative image inversion method ReStyle.

Concurrent few step diffusion image editing works Renoise and another method also calls TurboEdit.

This website is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. Website source code based on the Nerfies project page. If you want to reuse their source code, please credit them appropriately.