DeepProp: Extracting Deep Features from a Single Image for Edit Propagation

Yuki Endo¹, Satoshi Iizuka², Yoshihiro Kanamori¹, and Jun Mitani¹

¹University of Tsukuba
²Waseda University

Abstract:

Edit propagation is a technique that can propagate various image edits (e.g., colorization and recoloring) performed via user strokes to the entire image based on similarity of image features. In most previous work, users must manually determine the importance of each image feature (e.g., color, coordinates, and textures) in accordance with their needs and target images. We focus on representation learning that automatically learns feature representations only from user strokes in a single image instead of tuning existing features manually. To this end, this paper proposes an edit propagation method using a deep neural network (DNN). Our DNN, which consists of several layers such as convolutional layers and a feature combiner, extracts stroke-adapted visual features and spatial features, and then adjusts the importance of them. We also develop a learning algorithm for our DNN that does not suffer from the vanishing gradient problem, and hence avoids falling into undesirable locally optimal solutions. We demonstrate that edit propagation with deep features, without manual feature tuning, can achieve better results than previous work.

Our DNN Architecture:

Our model is a feedforward neural network that has four structures: visual feature extractor (VFE), spatial feature extractor (SFE), feature combiner (FC), and label estimator (LE). Unlike most previous methods of edit propagation, determining importance of each image feature is not required.

Publications:

1. Yuki Endo, Satoshi Iizuka, Yoshihiro Kanamori, and Jun Mitani, "DeepProp: Extracting Deep Features from a Single Image for Edit Propagation", Computer Graphics Forum (Proc. of Eurographics 2016), 35, 2, 189-201, 2016. [Pdf (64.7MB)]  [BibTex] [Code]