Citation

Jiang T, Gradus JL, Lash TL, Fox MP. Am. J. Epidemiol. 2021; 190(9): 1830-1840.

Copyright

(Copyright © 2021, Oxford University Press)

DOI

10.1093/aje/kwab010

PMID

33517416

PMCID

PMC8408353

Abstract

Although variables are often measured with error, the impact of measurement error on machine-learning predictions is seldom quantified. The purpose of this study was to assess the impact of measurement error on the performance of random-forest models and variable importance. First, we assessed the impact of misclassification (i.e., measurement error of categorical variables) of predictors on random-forest model performance (e.g., accuracy, sensitivity) and variable importance (mean decrease in accuracy) using data from the National Comorbidity Survey Replication (2001-2003). Second, we created simulated data sets in which we knew the true model performance and variable importance measures and could verify that quantitative bias analysis was recovering the truth in misclassified versions of the data sets. Our findings showed that measurement error in the data used to construct random forests can distort model performance and variable importance measures and that bias analysis can recover the correct results. This study highlights the utility of applying quantitative bias analysis in machine learning to quantify the impact of measurement error on study results.

Language: en

Keywords

Humans; Suicide, Attempted; Bias; Probability; machine learning; Machine Learning; noise; Computer Simulation; quantitative bias analysis; Datasets as Topic; measurement error; misclassification; random forests; Scientific Experimental Error