|
Abstract
|
The problem of missing data is common in research, and there is no exception for cognitive diagnostic assessment (CDA). Some studies have revealed that both the presence of missing values and the selection of different missing data processing methods would affect the results of CDA. Therefore, it is necessary to attach more attention to the problem in CDA and choose appropriate methods to deal with it. Although the problem in CDA has been explored before, previous studies did not consider multiple imputation (MI) and full information maximum likelihood (FIML), which are widely used in the field of missing data analysis. Moreover, previous studies neglected the comparison using empirical data and saturation models such as GDINA model. In summary, the main purpose of this study are to introduce MI and FIML into CDA, thus making a comprehensive comparison of different missing data handling methods, and further putting forward suggestions for handling missing data in practice.
Simulation study considered six factors: (1) Sample size: 200 participants, 400 participants, and 1000 participants; (2) Test length: 15 test items and 30 test items; (3) Quality of items: high quality, medium quality, and low quality; (4) Missing data mechanisms: missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR); (5) Missing rate: 10%, 20%, and 30%; (6) Missing data handling methods: zero replacement (ZR), MI-CART, MI-PMM, MI-LOGREG.BOOT, Expectation-Maximization algorithm (EM), and FIML. The GDINA model was used, and the analysis process was realized by the GDINA package in R software. Secondly, the PISA 2015 computer-based mathematics data were applied to compare the practical value of the proposed methods.
The results of simulation study revealed that: (1) Missing data results in a decrease in estimation accuracy. The absolute value of Bias and RMSE both increased and PCCR values of all methods decreased as the sample size, test length and the quality of the items decreased and the missing rate increased; (2) When estimating item parameters, EM performed best, followed by MI. Meanwhile, FIML and ZR methods were unstable; (3) When estimating the KS of participants, EM and FIML performed best as the missing data mechanism was MAR or MCAR. When the missing data mechanism was MNAR, EM, FIML and ZR performed best. The empirical study results further supported the simulation research results. It showed that: (1) For all empirical indicators, EM, FIML, and MI-PMM perform best on one or more indicators; (2) The results obtained under the empirical study and simulation study under the MNAR mechanism are very similar; (3) EM performs well on all indicators, and ZR and FIML methods are slightly worse than EM, followed by MI-PMM, LOGREG.BOOT and MI-CART.
In addition, based on the research results, the following suggestions were provided: (1) EM and FIML should be the first choice. However, if researchers do not want to get the complete data set, FIML could be used as a priority for missing data handling; (2) When the missing data mechanism was MAR or MCAR and the test length was not enough, researchers should avoid using the ZR method to deal with missing data. Finally, this paper ends with the prospects of future researches: (1) The multilevel scoring situation should also be studied; (2) The effectiveness of these methods should be tested in longitudinal research; (3) The performance of more methods of information matrix can be further compared in calculating the standard error to handle missing data; (4) Future research could focus on the missing mechanisms of data onto the real data.
Language: en |