Globally and Locally Consistent Image Completion
Satoshi Iizuka
Edgar Simo-Serra
Hiroshi Ishikawa
SIGGRAPH 2017
Abstract:
We present a novel approach for image completion that results in images that are both
locally and globally consistent. With a fully-convolutional neural network, we can
complete images of arbitrary resolutions by filling-in missing regions of any shape.
To train this image completion network to be consistent, we use global and local context
discriminators that are trained to distinguish real images from completed ones.
The global discriminator looks at the entire image to assess if it is coherent as a whole, while the
local discriminator looks only at a small area centered at the completed region
to ensure the local consistency of the generated patches.
The image completion network is then trained to fool the both context discriminator
networks, which requires it to generate images that are indistinguishable from real ones
with regard to overall consistency as well as in details.
We show that our approach can be used to complete a wide variety of scenes.
Furthermore, in contrast with the patch-based approaches such as PatchMatch, our
approach can generate fragments that do not appear elsewhere in the image,
which allows us to naturally complete the images of objects with familiar and highly specific
structures, such as faces.
Paper (6.9MB) Slide Code (GitHub) Supplemental (67.2MB) BibTex
Model Architecture:
Our architecture is composed of three networks: a completion network, a
global context discriminator, and a local context discriminator. The
completion network is fully convolutional and used to complete
the image, while both the global and the local context discriminators
are auxiliary networks used exclusively for training. These
discriminators are used to determine whether or not an image has
been completed consistently. The global discriminator takes the
full image as input to recognize global consistency of the scene,
while the local discriminator looks only at a small region around
the completed area in order to judge the quality of more detailed
appearance.
Comparison:
Comparison with Photoshop CS6 Content Aware Fill (PatchMatch [Barnes et al. 2009]), [Darabi et al. 2012], [Huang et al. 2014], and [Pathak et al. 2016] on a diverse set of scenes using random masks.
- Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. PatchMatch: A Randomized Correspondence Algorithm for Structural Image Editing.. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 28, 3 (2009), 24:1–24:11.
- Soheil Darabi, Eli Shechtman, Connelly Barnes, Dan B Goldman, and Pradeep Sen. Image Melding: Combining Inconsistent Images using Patch-based Synthesis.. ACM Transactions on Graphics (Proceedings of SIGGRAPH) 31, 4 (2012), 82:1–82:10.
- Jia-Bin Huang, Sing Bing Kang, Narendra Ahuja, and Johannes Kopf. Image Completion Using Planar Structure Guidance.. ACM Transactions on Graphics (Proceedings
of SIGGRAPH) 33, 4 (2014), 129:1-129:10.
-
Deepak Pathak, Philipp Krähenbühl, Jeff Donahue, Trevor Darrell, and Alexei Efros. Context Encoders: Feature Learning by Inpainting. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) 2016.
Additional results are here.
