Is it agentic enough? Benchmarking open models on your own tooling

Coding agents are increasingly interacting with our software, automating tasks from library selection to debugging. This shift introduces a new paradigm in library development: software must now be designed for effective agent interaction, not just correctness and speed. Clunky APIs and outdated documentation can lead agents down inefficient and more expensive paths.

Traditional benchmarks primarily evaluate final answers. However, this new approach focuses on the entire process—how much effort an agent expends to reach a solution. By examining models, library revisions, and tasks, this method measures not only correctness but also the efficiency of the pathway taken. The `transformers` library serves as a key case study for this benchmarking methodology.

Optimizing software for agents involves two core principles: discoverability and clarity. APIs must be clear, and documentation needs to be extensive, well-structured, and easily accessible with relevant examples. To ensure a tool works effectively for agents, it must be tested specifically for agentic use cases. This approach helps in identifying improvements that streamline agent workflows and reduce resource consumption.