Machine Learning-Based Fast Angular Prediction Mode Decision Technique in Video Coding

Sookyung Ryu, Jewon Kang

Research output: Contribution to journalArticlepeer-review

44 Scopus citations


In this paper, we propose a machine learning-based fast intra-prediction mode decision algorithm, using random forest that is an ensemble model of randomized decision trees. The random forest is used to estimate an intra-prediction mode from a prediction unit and to reduce encoding time significantly by avoiding the intensive Rate-Distortion (R-D) optimization of a number of intra-prediction modes. To this aim, we develop a randomized tree model including parameterized split functions at nodes to learn directional block-based features. The feature uses only four pixels reflecting a directional property of a block, and, thus the evaluation is fast and efficient. To integrate the proposed technique into the conventional video coding standard frameworks, the intra-prediction mode derived from the proposed technique, called an inferred mode, is used to shrink the pool of the candidate modes before carrying out the R-D optimization. The proposed technique is implemented into the high efficiency video coding test model reference software of the state-of-the-art video coding standard and joint exploration model reference software, by integrating the random forest trained off-line into the codecs. Experimental results demonstrate that the proposed technique achieves significant encoding time reduction with only slight coding loss as compared with the reference software models.

Original languageEnglish
Article number8412615
Pages (from-to)5525-5538
Number of pages14
JournalIEEE Transactions on Image Processing
Issue number11
StatePublished - Nov 2018

Bibliographical note

Publisher Copyright:
© 1992-2012 IEEE.


  • Fast intra prediction
  • HEVC test model (HM)
  • HEVC/H265
  • fast mode decision
  • joint exploration model (JEM)
  • machine learning
  • random forest


Dive into the research topics of 'Machine Learning-Based Fast Angular Prediction Mode Decision Technique in Video Coding'. Together they form a unique fingerprint.

Cite this