Learning Deeply Aggregated Alternating Minimization for General Inverse Problems

Hyungjoo Jung, Youngjung Kim, Dongbo Min, Hyunsung Jang, Namkoo Ha, Kwanghoon Sohn

Research output: Contribution to journalArticlepeer-review

3 Scopus citations


Regularization-based image restoration is one of the most powerful tools in image processing and computer vision thanks to its flexibility for handling various inverse problems. However, designing an optimal regularization function still remains unsolved since natural images and related scene types have a complex structure. In this paper, we present a general and principled framework, called deeply aggregated alternating minimization (DeepAM). We design a convolutional neural network (CNN) to implicitly parameterize the regularizer of the alternating minimization (AM) algorithm. Contrary to the conventional AM algorithm based on a point-wise proximal mapping, the DeepAM projects intermediate estimate into a set of natural images via deep aggregation. Since the CNN is fully integrated into the AM procedure, all parameters can be jointly optimized through end-to-end training. These properties enable the DeepAM to converge with a small number of iterations, while maintaining an algorithmic simplicity. We show that the DeepAM outperforms state-of-the-art methods, including nonlocal-based methods, Plug-and-Play regularization, and recent data-driven approaches. The effectiveness of our framework is demonstrated in a variety of image restoration tasks: Guassian denoising, deraining, deblurring, super-resolution, color-guided depth upsampling, and RGB/NIR restoration.

Original languageEnglish
Article number9146780
Pages (from-to)8012-8027
Number of pages16
JournalIEEE Transactions on Image Processing
StatePublished - 2020

Bibliographical note

Funding Information:
Manuscript received September 3, 2018; revised December 18, 2019; accepted July 9, 2020. Date of publication July 23, 2020; date of current version July 28, 2020. This work was supported by the Research and Development Program for Advanced Integrated-intelligence for Identification (AIID) through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT under Grant NRF-2018M3E3A1057289. The associate editor coordinating the review of this manuscript and approving it for publication was Prof. Xiaolin Wu. (Corresponding author: Kwanghoon Sohn.) Hyungjoo Jung and Kwanghoon Sohn are with the School of Electrical and Electronic Engineering, Yonsei University, Seoul 120-749, South Korea (e-mail: coolguy0220@yonsei.ac.kr; khsohn@yonsei.ac.kr).

Publisher Copyright:
© 1992-2012 IEEE.


  • Regularization-based image restoration
  • alternating minimization
  • convolutional neural network
  • half-quadratic minimization
  • joint restoration
  • proximal mapping


Dive into the research topics of 'Learning Deeply Aggregated Alternating Minimization for General Inverse Problems'. Together they form a unique fingerprint.

Cite this