Intelligent Data Cleaning for Data-centric AI: From Traditional Methods to Foundation Models—A Survey on Semantic-Driven Data Quality Enhancement in the Era of Large Language Models

Abstract

The concept of artificial intelligence has changed greatly from model center to data center. It has always been thought that the complexity of the algorithm is very important, but this idea is the upper limit of the performance of the model determined by the quality of data. However, there is noise in the real dataset, and the information is not enough and scattered. This new thinking has great problems. The method of cleaning up the previous data, although using fixed rules and simple statistics, is often unable to read complex semantic connections, and it is too numerous to cope with in the era of big data. Recently, the Large Language Models (LLMs) and Foundation Models came out, and the ability to understand and think about meaning was improved unprecedentedly, and new methods of managing data quality came into being. In this paper, from the point of view of data-centric AI, the technology of cleaning data is widely summarized. We use the rule-based system and machine learning to apply LLM to the most advanced methods of entity resolution, error detection, missing value imputation and other tasks, and then analyze the change of evaluation method from the previous accuracy index to the effect of downstream task. Hallucination, privacy preservation and computational cost of data cleaning using LLM are analyzed. Finally, the future research directions are listed, such as automated data curation, federated data cleaning, human-in-the-loop, etc., in order to build a powerful and intelligent data cleaning system.