TY - JOUR
T1 - Deep video prediction network-ased inter-frame coding in HEVC
AU - Lee, Jung Kyung
AU - Kim, Nayoung
AU - Cho, Seunghyun
AU - Kang, Je Won
N1 - Funding Information:
This work was supported by the Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korea government (MSIT) (Development of Audio/Video Coding and Light Field Media Fundamental Technologies for Ultra Realistic Tera-Media and Development of Compression and Transmission Technologies for Ultra High Quality Immersive Videos Supporting 6DoF) under Grant 2017-0-00072 and Grant 2018-0-00765.
Publisher Copyright:
© 2013 IEEE.
PY - 2020
Y1 - 2020
N2 - In this paper, we propose a novel Convolutional Neural Network (CNN) based video coding technique using a video prediction network (VPN) to support enhanced motion prediction in High Efficiency Video Coding (HEVC). Specifically, we design a CNN VPN to generate a virtual reference frame (VRF), which is synthesized using previously coded frames, to improve coding efficiency. The proposed VPN uses two sub-VPN architectures in cascade to predict the current frame in the same time instance. The VRF is expected to have higher temporal correlation than a conventional reference frame, and, thus it is substituted for a conventional reference frame. The proposed technique is incorporated into the HEVC inter-coding framework. Particularly, the VRF is managed in a HEVC reference picture list, so that each prediction unit (PU) can choose a better prediction signal through Rate-Distortion optimization without any additional side information. Furthermore, we modify the HEVC inter-prediction mechanisms of Advanced Motion Vector Prediction and Merge modes adaptively when the current PU uses the VRF as a reference frame. In this manner, the proposed technique can exploit the PU-wise multi-hypothesis prediction techniques in HEVC. Since the proposed VPN can perform both the video interpolation and extrapolation, it can be used for Random Access (RA) and Low Delay B (LD) coding configurations. It is shown in experimental results that the proposed technique provides -2.9% and -5.7% coding gains, respectively, in RA and LD coding configurations as compared to the HEVC reference software, HM 16.6 version.
AB - In this paper, we propose a novel Convolutional Neural Network (CNN) based video coding technique using a video prediction network (VPN) to support enhanced motion prediction in High Efficiency Video Coding (HEVC). Specifically, we design a CNN VPN to generate a virtual reference frame (VRF), which is synthesized using previously coded frames, to improve coding efficiency. The proposed VPN uses two sub-VPN architectures in cascade to predict the current frame in the same time instance. The VRF is expected to have higher temporal correlation than a conventional reference frame, and, thus it is substituted for a conventional reference frame. The proposed technique is incorporated into the HEVC inter-coding framework. Particularly, the VRF is managed in a HEVC reference picture list, so that each prediction unit (PU) can choose a better prediction signal through Rate-Distortion optimization without any additional side information. Furthermore, we modify the HEVC inter-prediction mechanisms of Advanced Motion Vector Prediction and Merge modes adaptively when the current PU uses the VRF as a reference frame. In this manner, the proposed technique can exploit the PU-wise multi-hypothesis prediction techniques in HEVC. Since the proposed VPN can perform both the video interpolation and extrapolation, it can be used for Random Access (RA) and Low Delay B (LD) coding configurations. It is shown in experimental results that the proposed technique provides -2.9% and -5.7% coding gains, respectively, in RA and LD coding configurations as compared to the HEVC reference software, HM 16.6 version.
KW - HEVC
KW - VVC
KW - Video coding
KW - convolutional neural network
KW - deep learning
KW - inter-prediction
KW - video prediction network
KW - virtual reference frame
UR - http://www.scopus.com/inward/record.url?scp=85086065379&partnerID=8YFLogxK
U2 - 10.1109/ACCESS.2020.2993566
DO - 10.1109/ACCESS.2020.2993566
M3 - Article
AN - SCOPUS:85086065379
VL - 8
SP - 95906
EP - 95917
JO - IEEE Access
JF - IEEE Access
SN - 2169-3536
M1 - 9090881
ER -