Position: Let's Develop Data Probes to Fundamentally Understand How Data Affects LLM Performance

A new position paper calls for the creation of "data probes" to thoroughly understand the relationship between data and the performance of Large Language Models (LLMs). This research is considered critical for improving LLM capabilities and fostering responsible AI development. The paper, titled "Position: Let's Develop Data Probes to Fundamentally Understand How Data Affects LLM Performance," was authored by Shiqiang Wang and three co-authors. It was submitted on May 11, 2026. The full text is available via a PDF link. This work highlights the growing need for tools that can analyze how various data characteristics impact LLM outcomes. Tools like alphaXiv, CatalyzeX Code Finder, DagsHub, GotitPub, Hugging Face, and ScienceCast are available for code, data, and media associated with such research. Additionally, platforms like Replicate, Hugging Face Spaces, and TXYZ.AI offer demo functionalities. The paper also references various bibliographic and citation tools, including NASA ADS, Google Scholar, Semantic Scholar, and BibTeX. Recommenders and search tools such as Influence Flower, CORE Recommender, and those provided by arXivLabs further support research in this area. arXivLabs, an initiative that allows collaborators to develop and share new features on the arXiv website, emphasizes values of openness, community, excellence, and user data privacy. They partner with organizations that adhere to these principles.