Performance Evaluation of Image Quality Metrics for Perceptual Assessment of Low-dose Computed Tomography Images

Wonkyeong Lee; Eunbyeol Cho; Wonjin Kim; Jang Hwan Choi

doi:10.1117/12.2612541

Performance Evaluation of Image Quality Metrics for Perceptual Assessment of Low-dose Computed Tomography Images

Wonkyeong Lee, Eunbyeol Cho, Wonjin Kim, Jang Hwan Choi

Department of Mechanical & Biomedical Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

Objective image quality metrics (IQMs) are widely developed and utilized, considering that they can lead to optimal radiation doses in computed tomography (CT) imaging. However, how well these IQMs relate to a radiologist s perception of subjective image quality, which is the gold standard for assessing diagnostic image quality, has not been fully explored. Therefore, in this study, we aim to analyze the relationship between subjective and objective quality metrics. We compared 13 full-reference and no-reference IQMs, including root mean square error, peak signal-To-noise ratio, structural similarity index (SSIM), multi-scale SSIM, information content weighted (IW)-SSIM, gradient magnitude similarity deviation, feature similarity index, noise quality metric, visual information fidelity, natural image quality evaluator, blind/referenceless image spatial quality evaluator, perception-based image quality evaluator, and the model observer nonpre-whitening with eye filter (NPWE). The data used in this study were CT images under seven noise levels. The scores obtained from these data with the objective IQMs were then compared with the three radiologists scores by using Pearson linear correlation coefficient (PLCC) and Spearman s rank order correlation coefficient (SROCC). The results show that SSIM performs best in terms of PLCC and SROCC but lacked some characteristics of the radiologists assessment. Fullreference IQMs, except for IW-SSIM, generally outperform no-reference IQMs. No-reference IQMs show poor PLCC and SROCC scores, and the model observer NPWE shows the worst performance among them. These results may contribute to evaluating and developing IQMs with the preferences of radiologists. 2022 SPIE.

Original language	English
Title of host publication	Medical Imaging 2022
Subtitle of host publication	Image Perception, Observer Performance, and Technology Assessment
Editors	Claudia R. Mello-Thoms, Claudia R. Mello-Thoms, Sian Taylor-Phillips
Publisher	SPIE
ISBN (Electronic)	9781510649453
DOIs	https://doi.org/10.1117/12.2612541
State	Published - 2022
Event	Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment - Virtual, Online Duration: 21 Mar 2022 → 27 Mar 2022

Publication series

Name	Progress in Biomedical Optics and Imaging - Proceedings of SPIE
Volume	12035
ISSN (Print)	1605-7422

Conference

Conference	Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment
City	Virtual, Online
Period	21/03/22 → 27/03/22

Bibliographical note

Funding Information:
This work was partly supported by the Technology development Program of MSS [S3146559], the National Research Foundation of Korea (NRF) grants funded by the Korean government (MSIP: Ministry of Science, ICT, and Future Planning; No. NRF-2020R1A4A1016619, NRF-2020R1F1A1073774), and the Korea Medical Device Development Fund grant funded by the Korea government (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health & Welfare, the Ministry of Food and Drug Safety) (Project Number: KMDF_PR_20200901_0016, 9991006689). The funders had no role in study design, data collection and analysis, decision to publish, or manuscript preparation.

Publisher Copyright:
© 2022 SPIE. All rights reserved.

Keywords

image quality assessment
image quality metrics
low-dose CT image
objective assessment
subject assessment

Access to Document

10.1117/12.2612541

Cite this

Lee, W., Cho, E., Kim, W., & Choi, J. H. (2022). Performance Evaluation of Image Quality Metrics for Perceptual Assessment of Low-dose Computed Tomography Images. In C. R. Mello-Thoms, C. R. Mello-Thoms, & S. Taylor-Phillips (Eds.), Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment Article 1203515 (Progress in Biomedical Optics and Imaging - Proceedings of SPIE; Vol. 12035). SPIE. https://doi.org/10.1117/12.2612541

Lee, Wonkyeong ; Cho, Eunbyeol ; Kim, Wonjin et al. / Performance Evaluation of Image Quality Metrics for Perceptual Assessment of Low-dose Computed Tomography Images. Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment. editor / Claudia R. Mello-Thoms ; Claudia R. Mello-Thoms ; Sian Taylor-Phillips. SPIE, 2022. (Progress in Biomedical Optics and Imaging - Proceedings of SPIE).

@inproceedings{4f00a6e676f34fa191b54cadbb646a1a,

title = "Performance Evaluation of Image Quality Metrics for Perceptual Assessment of Low-dose Computed Tomography Images",

abstract = "Objective image quality metrics (IQMs) are widely developed and utilized, considering that they can lead to optimal radiation doses in computed tomography (CT) imaging. However, how well these IQMs relate to a radiologist s perception of subjective image quality, which is the gold standard for assessing diagnostic image quality, has not been fully explored. Therefore, in this study, we aim to analyze the relationship between subjective and objective quality metrics. We compared 13 full-reference and no-reference IQMs, including root mean square error, peak signal-To-noise ratio, structural similarity index (SSIM), multi-scale SSIM, information content weighted (IW)-SSIM, gradient magnitude similarity deviation, feature similarity index, noise quality metric, visual information fidelity, natural image quality evaluator, blind/referenceless image spatial quality evaluator, perception-based image quality evaluator, and the model observer nonpre-whitening with eye filter (NPWE). The data used in this study were CT images under seven noise levels. The scores obtained from these data with the objective IQMs were then compared with the three radiologists scores by using Pearson linear correlation coefficient (PLCC) and Spearman s rank order correlation coefficient (SROCC). The results show that SSIM performs best in terms of PLCC and SROCC but lacked some characteristics of the radiologists assessment. Fullreference IQMs, except for IW-SSIM, generally outperform no-reference IQMs. No-reference IQMs show poor PLCC and SROCC scores, and the model observer NPWE shows the worst performance among them. These results may contribute to evaluating and developing IQMs with the preferences of radiologists. 2022 SPIE.",

keywords = "image quality assessment, image quality metrics, low-dose CT image, objective assessment, subject assessment",

author = "Wonkyeong Lee and Eunbyeol Cho and Wonjin Kim and Choi, {Jang Hwan}",

note = "Funding Information: This work was partly supported by the Technology development Program of MSS [S3146559], the National Research Foundation of Korea (NRF) grants funded by the Korean government (MSIP: Ministry of Science, ICT, and Future Planning; No. NRF-2020R1A4A1016619, NRF-2020R1F1A1073774), and the Korea Medical Device Development Fund grant funded by the Korea government (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health & Welfare, the Ministry of Food and Drug Safety) (Project Number: KMDF_PR_20200901_0016, 9991006689). The funders had no role in study design, data collection and analysis, decision to publish, or manuscript preparation. Publisher Copyright: {\textcopyright} 2022 SPIE. All rights reserved.; Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment ; Conference date: 21-03-2022 Through 27-03-2022",

year = "2022",

doi = "10.1117/12.2612541",

language = "English",

series = "Progress in Biomedical Optics and Imaging - Proceedings of SPIE",

publisher = "SPIE",

editor = "Mello-Thoms, {Claudia R.} and Mello-Thoms, {Claudia R.} and Sian Taylor-Phillips",

booktitle = "Medical Imaging 2022",

}

Lee, W, Cho, E, Kim, W & Choi, JH 2022, Performance Evaluation of Image Quality Metrics for Perceptual Assessment of Low-dose Computed Tomography Images. in CR Mello-Thoms, CR Mello-Thoms & S Taylor-Phillips (eds), Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment., 1203515, Progress in Biomedical Optics and Imaging - Proceedings of SPIE, vol. 12035, SPIE, Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment, Virtual, Online, 21/03/22. https://doi.org/10.1117/12.2612541

Performance Evaluation of Image Quality Metrics for Perceptual Assessment of Low-dose Computed Tomography Images. / Lee, Wonkyeong; Cho, Eunbyeol; Kim, Wonjin et al.
Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment. ed. / Claudia R. Mello-Thoms; Claudia R. Mello-Thoms; Sian Taylor-Phillips. SPIE, 2022. 1203515 (Progress in Biomedical Optics and Imaging - Proceedings of SPIE; Vol. 12035).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Performance Evaluation of Image Quality Metrics for Perceptual Assessment of Low-dose Computed Tomography Images

AU - Lee, Wonkyeong

AU - Cho, Eunbyeol

AU - Kim, Wonjin

AU - Choi, Jang Hwan

N1 - Funding Information: This work was partly supported by the Technology development Program of MSS [S3146559], the National Research Foundation of Korea (NRF) grants funded by the Korean government (MSIP: Ministry of Science, ICT, and Future Planning; No. NRF-2020R1A4A1016619, NRF-2020R1F1A1073774), and the Korea Medical Device Development Fund grant funded by the Korea government (the Ministry of Science and ICT, the Ministry of Trade, Industry and Energy, the Ministry of Health & Welfare, the Ministry of Food and Drug Safety) (Project Number: KMDF_PR_20200901_0016, 9991006689). The funders had no role in study design, data collection and analysis, decision to publish, or manuscript preparation. Publisher Copyright: © 2022 SPIE. All rights reserved.

PY - 2022

Y1 - 2022

N2 - Objective image quality metrics (IQMs) are widely developed and utilized, considering that they can lead to optimal radiation doses in computed tomography (CT) imaging. However, how well these IQMs relate to a radiologist s perception of subjective image quality, which is the gold standard for assessing diagnostic image quality, has not been fully explored. Therefore, in this study, we aim to analyze the relationship between subjective and objective quality metrics. We compared 13 full-reference and no-reference IQMs, including root mean square error, peak signal-To-noise ratio, structural similarity index (SSIM), multi-scale SSIM, information content weighted (IW)-SSIM, gradient magnitude similarity deviation, feature similarity index, noise quality metric, visual information fidelity, natural image quality evaluator, blind/referenceless image spatial quality evaluator, perception-based image quality evaluator, and the model observer nonpre-whitening with eye filter (NPWE). The data used in this study were CT images under seven noise levels. The scores obtained from these data with the objective IQMs were then compared with the three radiologists scores by using Pearson linear correlation coefficient (PLCC) and Spearman s rank order correlation coefficient (SROCC). The results show that SSIM performs best in terms of PLCC and SROCC but lacked some characteristics of the radiologists assessment. Fullreference IQMs, except for IW-SSIM, generally outperform no-reference IQMs. No-reference IQMs show poor PLCC and SROCC scores, and the model observer NPWE shows the worst performance among them. These results may contribute to evaluating and developing IQMs with the preferences of radiologists. 2022 SPIE.

AB - Objective image quality metrics (IQMs) are widely developed and utilized, considering that they can lead to optimal radiation doses in computed tomography (CT) imaging. However, how well these IQMs relate to a radiologist s perception of subjective image quality, which is the gold standard for assessing diagnostic image quality, has not been fully explored. Therefore, in this study, we aim to analyze the relationship between subjective and objective quality metrics. We compared 13 full-reference and no-reference IQMs, including root mean square error, peak signal-To-noise ratio, structural similarity index (SSIM), multi-scale SSIM, information content weighted (IW)-SSIM, gradient magnitude similarity deviation, feature similarity index, noise quality metric, visual information fidelity, natural image quality evaluator, blind/referenceless image spatial quality evaluator, perception-based image quality evaluator, and the model observer nonpre-whitening with eye filter (NPWE). The data used in this study were CT images under seven noise levels. The scores obtained from these data with the objective IQMs were then compared with the three radiologists scores by using Pearson linear correlation coefficient (PLCC) and Spearman s rank order correlation coefficient (SROCC). The results show that SSIM performs best in terms of PLCC and SROCC but lacked some characteristics of the radiologists assessment. Fullreference IQMs, except for IW-SSIM, generally outperform no-reference IQMs. No-reference IQMs show poor PLCC and SROCC scores, and the model observer NPWE shows the worst performance among them. These results may contribute to evaluating and developing IQMs with the preferences of radiologists. 2022 SPIE.

KW - image quality assessment

KW - image quality metrics

KW - low-dose CT image

KW - objective assessment

KW - subject assessment

UR - http://www.scopus.com/inward/record.url?scp=85131870590&partnerID=8YFLogxK

U2 - 10.1117/12.2612541

DO - 10.1117/12.2612541

M3 - Conference contribution

AN - SCOPUS:85131870590

T3 - Progress in Biomedical Optics and Imaging - Proceedings of SPIE

BT - Medical Imaging 2022

A2 - Mello-Thoms, Claudia R.

A2 - Taylor-Phillips, Sian

PB - SPIE

T2 - Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment

Y2 - 21 March 2022 through 27 March 2022

ER -

Lee W, Cho E, Kim W, Choi JH. Performance Evaluation of Image Quality Metrics for Perceptual Assessment of Low-dose Computed Tomography Images. In Mello-Thoms CR, Mello-Thoms CR, Taylor-Phillips S, editors, Medical Imaging 2022: Image Perception, Observer Performance, and Technology Assessment. SPIE. 2022. 1203515. (Progress in Biomedical Optics and Imaging - Proceedings of SPIE). doi: 10.1117/12.2612541

Performance Evaluation of Image Quality Metrics for Perceptual Assessment of Low-dose Computed Tomography Images

Abstract

Publication series

Conference

Bibliographical note

Keywords

Access to Document

Fingerprint

Cite this