Machine learning-based automatic classification of video recorded neonatal manipulations and associated physiological parameters: A feasibility study

Harpreet Singh, Satoshi Kusuda, Ryan M. McAdams, Shubham Gupta, Jayant Kalra, Ravneet Kaur, Ritu Das, Saket Anand, Ashish Kumar Pandey, Su Jin Cho, Satish Saluja, Justin J. Boutilier, Suchi Saria, Jonathan Palma, Avneet Kaur, Gautam Yadav, Yao Sun

Research output: Contribution to journalArticlepeer-review

4 Scopus citations


Our objective in this study was to determine if machine learning (ML) can automatically recognize neonatal manipulations, along with associated changes in physiological parameters. A retrospective observational study was carried out in two Neonatal Intensive Care Units (NICUs) between December 2019 to April 2020. Both the video and physiological data (heart rate (HR) and oxygen saturation (SpO2 )) were captured during NICU hospitalization. The proposed classification of neonatal manipulations was achieved by a deep learning system consisting of an Inception-v3 convolutional neural network (CNN), followed by transfer learning layers of Long Short-Term Memory (LSTM). Physiological signals prior to manipulations (baseline) were compared to during and after manipulations. The validation of the system was done using the leave-one-out strategy with input of 8 s of video exhibiting manipulation activity. Ten neonates were video recorded during an average length of stay of 24.5 days. Each neonate had an average of 528 manipulations during their NICU hospitalization, with the average duration of performing these manipulations varying from 28.9 s for patting, 45.5 s for a diaper change, and 108.9 s for tube feeding. The accuracy of the system was 95% for training and 85% for the validation dataset. In neonates <32 weeks’ gestation, diaper changes were associated with significant changes in HR and SpO2, and, for neonates ≥32 weeks’ gestation, patting and tube feeding were associated with significant changes in HR. The presented system can classify and document the manipulations with high accuracy. Moreover, the study suggests that manipulations impact physiological parameters.

Original languageEnglish
Article number1
Issue number1
StatePublished - Jan 2021

Bibliographical note

Publisher Copyright:
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license ( licenses/by/4.0/).


  • CNN
  • Electronic medical records
  • IoT
  • LSTM
  • Machine learning
  • Neonatal intensive care units
  • Physiological deviations
  • Physiological parameters
  • Streaming server
  • Video monitoring


Dive into the research topics of 'Machine learning-based automatic classification of video recorded neonatal manipulations and associated physiological parameters: A feasibility study'. Together they form a unique fingerprint.

Cite this