TY - GEN
T1 - Super-resolution of Multi-view ERP 360-Degree Images with Two-Stage Disparity Refinement
AU - Kim, Hee Jae
AU - Kang, Je Won
AU - Lee, Byung Uk
N1 - Funding Information:
ACKNOWLEDGMENT This work was supported by Institute of Information and communications Technology Planning and Evaluation (IITP) grant funded by the Korea government (MSIT) (No.2018-0-00765 Development of Compression and Transmission Technologies for Ultra High Quality Immersive Videos Supporting 6DoF) and the National Research Foundation of Korea (NRF) (No.NRF-2019R1C1C1010249)
Publisher Copyright:
© 2020 APSIPA.
PY - 2020/12/7
Y1 - 2020/12/7
N2 - In this paper, we propose a novel super-resolution (SR) technique for multi-view 360-degree images in equirectangular projection (ERP) format. To the best of our knowledge, the proposed algorithm is the first study of multi-view 360-degree images in ERP. In multi-view SR (MV-SR), it is important to fuse the knowledge of features at different viewpoints, but the task is hardly achieved using a conventional CNN because conventional convolution is shift invariant. Thus, to solve the problem, we take a coarse-to-fine approach to exploit the correlation among multi-views in an ERP domain. First, we conduct depth-based warping on reference ERP to synthesize the image with the same viewpoint of the target low-resolution (LR) ERP. The non-linear distortion between the two ERP images can be remarkably reduced after the proposed warping. Second, we employ a flow estimator to refine the remaining flow between the warped reference image and the LR image. Our CNN architecture generates the SR at the end of the network by combining the features of LR-ERP and the warped reference ERP. It is demonstrated with experimental results that the proposed algorithm provides significantly improved quality of multi-view 360-degree images for SR as compared to the state-of-the-art in MV-SR.
AB - In this paper, we propose a novel super-resolution (SR) technique for multi-view 360-degree images in equirectangular projection (ERP) format. To the best of our knowledge, the proposed algorithm is the first study of multi-view 360-degree images in ERP. In multi-view SR (MV-SR), it is important to fuse the knowledge of features at different viewpoints, but the task is hardly achieved using a conventional CNN because conventional convolution is shift invariant. Thus, to solve the problem, we take a coarse-to-fine approach to exploit the correlation among multi-views in an ERP domain. First, we conduct depth-based warping on reference ERP to synthesize the image with the same viewpoint of the target low-resolution (LR) ERP. The non-linear distortion between the two ERP images can be remarkably reduced after the proposed warping. Second, we employ a flow estimator to refine the remaining flow between the warped reference image and the LR image. Our CNN architecture generates the SR at the end of the network by combining the features of LR-ERP and the warped reference ERP. It is demonstrated with experimental results that the proposed algorithm provides significantly improved quality of multi-view 360-degree images for SR as compared to the state-of-the-art in MV-SR.
UR - http://www.scopus.com/inward/record.url?scp=85100941966&partnerID=8YFLogxK
M3 - Conference contribution
AN - SCOPUS:85100941966
T3 - 2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2020 - Proceedings
SP - 1283
EP - 1286
BT - 2020 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference, APSIPA ASC 2020 - Proceedings
PB - Institute of Electrical and Electronics Engineers Inc.
Y2 - 7 December 2020 through 10 December 2020
ER -