Citation

Lee JT, Thakuriah P. J. Transp. Res. Forum 2004; 43(2): 37-52.

Copyright

(Copyright © 2004, Transportation Research Forum)

DOI

unavailable

PMID

unavailable

Abstract

Databases from various sources often need to be merged in order to obtain necessary pieces of information for transportation policy development. However, diverse databases often lack unique identifiers or have data quality problems. This paper investigates a probabilistic linkage method as a potential solution to overcome data quality problems in the context of linking databases in the commercial motor vehicle and carrier sector. The method is demonstrated by linking commercial motor vehicle inspection files kept by the Illinois State Police and the inspection files available from the Illinois portion of the Motor Carrier Management Information System. Since one of the files to be matched is a subset of the other and there is a relational unique identifier, the application allows us to validate the methodology. The results show 6,228 correct identifications of true matched record pairs out of 6,335 actual true matches (more than 99%) between the two files. The number of erroneously identified record pairs is 690 (about 11% of the actual true matched pairs.) About 1,540 record pairs were generated to be manually reviewed for correct identification. Sensitivity analysis is conducted of error rates with respect to variations in the optimal thresholds for merging the databases. In this application, a simple cost estimate for hiring a clerk to review record pairs in the uncertain region was used to determine optimum thresholds.