Layered Skill Test Coverage: A Better Way to Evaluate AI Agent Skills

AI agents are becoming more capable, and skills are an important part of that progress. Skills give agents instructions, workflows, and knowledge to complete specific tasks. But how do we know if a skill is actually useful?

In my paper, Layered Skill Coverage: Telling Never Saw It Apart from Saw It and Ignored It, I explore a simple problem with the way we evaluate agent skills.

Traditional software testing uses coverage metrics to understand which lines of code or functions have been tested. A similar idea can help us evaluate AI agent skills, but there is an important difference.

An agent might never discover a skill. It might discover the skill but decide not to use it. Or it might use the skill and still fail to follow its instructions properly. Treating all these situations the same can hide important problems.

My research introduces Layered Skill Coverage to help distinguish these cases. The goal is to understand whether a skill was available to the agent, whether the agent retrieved it, and whether it followed the instructions.

This distinction matters because simply counting successful tasks does not tell us why a skill worked or failed. Better measurements can help developers identify missing instructions, improve skill design, and understand how agents use the tools available to them.

I believe evaluating agent skills deserves the same careful attention that software engineers give to testing their code. As AI agents become part of real software development workflows, we need better ways to measure not just their results, but also how they use the instructions we provide.

This paper is an attempt to take a step in that direction.

Read the paper: Layered Skill Coverage on Zenodo