Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification

Satoshi Iizuka* Edgar Simo-Serra* Hiroshi Ishikawa (*equal contribution)

SIGGRAPH 2016


Colorado National Park, 1941	Textile Mill, June 1937	Berry Field, June 1909	Hamilton, 1936

Abstract:

We present a novel technique to automatically colorize grayscale images that combines both global priors and local image features. Based on Convolutional Neural Networks, our deep network features a fusion layer that allows us to elegantly merge local information dependent on small image patches with global priors computed using the entire image. The entire framework, including the global and local priors as well as the colorization model, is trained in an end-to-end fashion. Furthermore, our architecture can process images of any resolution, unlike most existing approaches based on CNN. We leverage an existing large-scale scene classification database to train our model, exploiting the class labels of the dataset to more efficiently and discriminatively learn the global priors. We validate our approach with a user study and compare against the state of the art, where we show significant improvements. Furthermore, we demonstrate our method extensively on many different types of images, including black-and-white photography from over a hundred years ago, and show realistic colorizations.

Paper (15.3MB) Slide (5.3MB) Online demo Code (GitHub) BibTex

Colorization Architecture:

Our model consists of four main components: a low-level features network, a mid-level features network, a global features network, and a colorization network. The components are all tightly coupled and trained in an end-to-end fashion. The output of our model is the chrominance of the image which is fused with the luminance to form the output image.

Comparison：

Comparison against two related approaches using roughly a hundred year old black-and-white photography from the US National Archives.

Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning Representations for Automatic Colorization. In ECCV 2016.
Richard Zhang, Phillip Isola, and Alexei A. Efros. Colorful Image Colorization. In ECCV 2016.

Input	[Larsson et al. 2016]	[Zhang et al. 2016]	Ours

Additional results are here.

Results:

Additional results are here.

Publication:

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa.
"Let there be Color!: Joint End-to-end Learning of Global and Local Image Priors for Automatic Image Colorization with Simultaneous Classification".
ACM Transaction on Graphics (Proc. of SIGGRAPH), 35(4):110, 2016.

@Article{IizukaSIGGRAPH2016,

      author =
    {Satoshi Iizuka and Edgar Simo-Serra and Hiroshi Ishikawa},

      title = {{Let there be Color!: Joint End-to-end
    Learning of Global and Local Image Priors for Automatic Image
    Colorization with Simultaneous Classification}},

      journal = "ACM Transactions on Graphics
    (Proc. of SIGGRAPH 2016)",

      year = 2016,

      volume = 35,

      number = 4,

      pages = 110:1--110:11,

      articleno = 110,

    }

This work was partially supported by JST CREST.