In-N-Out: Face Video Inversion and Editing with Volumetric Decomposition
Adobe Research
Adobe Research
3D-aware GANs offer new capabilities for creative content editing, such as view synthesis, while preserving the editing capability of their 2D counterparts. Using GAN inversion, these methods can reconstruct an image or a video by optimizing/predicting a latent code and achieve semantic editing by manipulating the latent code. However, a model pre-trained on a face dataset (e.g., FFHQ) often has difficulty handling faces with out-of-distribution (OOD) objects, (e.g., heavy make-up or occlusions). We address this issue by explicitly modeling OOD objects in face videos. Our core idea is to represent the face in a video using two neural radiance fields, one for in-distribution and the other for out-of-distribution data, and compose them together for reconstruction. Such explicit decomposition alleviates the inherent trade-off between reconstruction fidelity and editability. We evaluate our method's reconstruction accuracy and editability on challenging real videos and showcase favorable results against other baselines.
Our method can be applied to diverse videos from the Internet. We show reconstruction results below.
We can edit it with available semantic editing methods, e.g., InterfaceGAN, StyleCLIP.
After reconstruction, we can acquire novel views.
By setting the weight of OOD pixels to 0, we can remove the OOD object.
@article{xu2023video3deditgan,
author = {Xu, Yiran and Shu, Zhixin and Smith, Cameron and Huang, Jia-Bin and Oh, Seoung Wug},
title = {In-N-Out: Face Video Inversion and Editing with Volumetric Decomposition},
journal = {arXiv preprint arXiv: 2302.04871},
year = {2023},
}