Dynamic motion estimation and evolution video prediction network

Nayoung Kim, Je Won Kang

Research output: Contribution to journalArticlepeer-review

9 Scopus citations

Abstract

Future video prediction provides valuable information that helps a computer machine understand the surrounding environment and make critical decisions in real-time. However, long-term video prediction remains a challenging problem due to the complicated spatiotemporal dynamics in a video. In this paper, we propose a dynamic motion estimation and evolution (DMEE) network model to generate unseen future videos from the observed videos in the past. Our primary contribution is to use trained kernels in convolutional neural network (CNN) and long short-term memory (LSTM) architectures, adapted to each time step and sample position, to efficiently manage spatiotemporal dynamics. DMEE uses the motion estimation (ME) and motion update (MU) kernels to predict the future video using an end-to-end prediction-update process. In the prediction, the ME kernel estimates the temporal changes. In the update step, the MU kernel combines the estimates with the previously generated frames as reference frames using a weighted average. The kernels are not only used for a current frame, but also are evolved to generate successive frames to enable temporally specific filtering. We perform qualitative performance analysis and quantitative performance analysis based on the peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and video classification score developed for examining the visual quality of the generated video. It is demonstrated with experiments that our algorithm provides better qualitative and quantitative performance superior to the current state-of-the-art algorithms. Our source codes are available in https://github.com/Nayoung-Kim-ICP/Video-Generation.

Original languageEnglish
Pages (from-to)3986-3998
Number of pages13
JournalIEEE Transactions on Multimedia
Volume23
DOIs
StatePublished - 2021

Bibliographical note

Publisher Copyright:
© 1999-2012 IEEE.

Keywords

  • Convolutional Neural Network
  • Deep learning
  • Long Short-term Memory
  • Long-term video generation and prediction
  • Video understanding and analysis

Fingerprint

Dive into the research topics of 'Dynamic motion estimation and evolution video prediction network'. Together they form a unique fingerprint.

Cite this