Super-resolution of Multi-view ERP 360-Degree Images with Two-Stage Disparity Refinement

In this paper, we propose a novel super-resolution (SR) technique for multi-view 360-degree images in equirectangular projection (ERP) format. To the best of our knowledge, the proposed algorithm is the first study of multi-view 360-degree images in ERP. In multi-view SR (MV-SR), it is important to fuse the knowledge of features at different viewpoints, but the task is hardly achieved using a conventional CNN because conventional convolution is shift invariant. Thus, to solve the problem, we take a coarse-to-fine approach to exploit the correlation among multi-views in an ERP domain. First, we conduct depth-based warping on reference ERP to synthesize the image with the same viewpoint of the target low-resolution (LR) ERP. The non-linear distortion between the two ERP images can be remarkably reduced after the proposed warping. Second, we employ a flow estimator to refine the remaining flow between the warped reference image and the LR image. Our CNN architecture generates the SR at the end of the network by combining the features of LR-ERP and the warped reference ERP. It is demonstrated with experimental results that the proposed algorithm provides significantly improved quality of multi-view 360-degree images for SR as compared to the state-of-the-art in MV-SR.