Agent skill guide
Evaluate an Agent Skill Workflow: Baselines, Artifacts and Acceptance Checks
Build a controlled skill trial with synthetic fixtures, observable acceptance checks, a no-skill baseline and a downloadable results ledger. Distinguish discovery from verified task quality.
Evaluate an agent skill by checking the actual artifact against predefined criteria and comparing it with the same task performed without the skill. Keep environment, input and authorized tools stable. This guide, reviewed on 20 October 2026, provides a proposed evaluation procedure and downloadable ledger. The scenarios below are test designs; they are not benchmark results or reports of executed third-party skills.
What counts as success at each stage?
| Stage | Evidence | What remains unproven |
|---|---|---|
| Discovered | The host lists the intended skill | Its helpers, tools and permissions work |
| Executed | The agent followed the workflow and produced an output | The output is correct or useful |
| Accepted for the fixture | The artifact passes the stated checks | Reliability on other inputs or in production |
| Useful in repeated work | Representative trials show acceptable correction effort, quality and cost | Future versions will behave identically |
Keep a separate result for each stage. “The command completed” is meaningful execution evidence but cannot substitute for reading a report, checking a formula or verifying a browser’s final state. If a check could not run, record it as not run rather than passed.
How do you set up a fair baseline?
Write one task brief and use the same fixture, host, model and available tools for the baseline and the skill-assisted trial. Start with clean task contexts so an earlier answer does not leak into the comparison. Record which skill folders were active: an implicit skill invocation can contaminate a supposed no-skill baseline.
Define acceptance before seeing outputs. Choose a small representative set that includes an ordinary case, a boundary case and a case the skill should decline or route elsewhere. This is an editorial starting design, not a statistically sufficient sample size. Expand it when the cost of failure warrants more evidence.
Repeat trials when results vary, and preserve individual runs instead of reporting only the best one. If one trial uses more capable tools or receives extra corrective prompting, disclose that difference rather than attributing the result entirely to the skill.
What does a document-to-decision workflow look like?
Use a synthetic project CSV to create a short notebook, a one-page report and an internal update. This combines three distinct artifacts without requiring a connected production account. Each stage needs its own contract; otherwise an error can travel through the whole chain while every file still looks plausible.
- Notebook stage: use the Jupyter Notebook profile to organize the analysis. Check that totals match an independent calculation and that the notebook can run from a fresh kernel when execution is authorized.
- Report stage: use the PDF profile to turn approved values into a one-page report. Check numbers again, render every page and inspect clipping and readability.
- Communication stage: use the Internal Communications profile to draft an update from the accepted report. Check that the message introduces no new completion claim, date or owner.
These source profiles come from different publisher ecosystems. Combining them is a proposed process design; it is not a claim that one host can run all three unchanged. You can perform the stages in separate approved environments or keep the final communication step as a human-reviewed draft.
Which synthetic fixture can you reproduce?
Download the three-project synthetic CSV. Atlas has 12 planned and 10 actual hours, Beacon has 8 planned and 11 actual hours, and Cedar has 10 planned and 10 actual hours. The expected totals are 30 planned hours and 31 actual hours, so actual minus planned is +1 hour. These are arithmetic expectations for an invented fixture, not observed skill performance.
Ask for a notebook that preserves all three rows, a one-page PDF showing the same totals and an update that reports the one-hour aggregate variance without inventing completion dates or causes. A useful boundary case is an empty actual-hours cell: decide in advance whether the workflow must stop or explicitly mark missing data. It must not silently turn an unknown value into a factual zero.
Which acceptance checks should you write?
| Artifact | Concrete check | Failure example |
|---|---|---|
| Notebook | Recalculate a known fixture total independently | Silent type conversion drops a row |
| Compare source values and inspect all rendered pages | A correct value is clipped from the visible table | |
| Research brief | Follow each consequential citation to the source | A source URL exists but does not support the claim |
| Browser task | Verify the requested final state and intended data change | A click occurs but validation prevents submission |
| Code repair | Reproduce the reported failure, then rerun relevant checks | A local test passes while the remote check still fails |
| Marketing copy | Trace every quantitative claim to approved evidence | A source’s example metric becomes a fictional product result |
Keep critical failures as gates. An unauthorized data transfer or an incorrect total should not disappear inside a favorable average quality score. For softer qualities such as clarity, define a rubric and have the intended reader review anonymized outputs when practical.
How should you record time, corrections and cost?
Track observed task time, the manual effort needed to repair the output, tool calls and available usage cost. Label estimates separately from billed or observed values. A shorter first draft can still cost more if a reviewer must recheck every citation or rebuild a broken artifact.
The evaluation ledger CSV contains headers for recording actual runs, not simulated scores. Give each run a stable ID and attach paths to its input fixture, output and review evidence. Never store API keys, private account identifiers or raw confidential prompts in a shareable ledger.
How do failures become improvements?
Classify a failure before changing the skill. Missing tools belong to environment setup; wrong input assumptions belong to the task contract; stale external rules belong to dependency tracking; unsupported claims belong to output review. Change one relevant variable, rerun the same fixture and keep a held-out case to detect overfitting.
For skill authors, the Skill Creator profile describes an evaluation-led iteration process. Its helper scripts and repeated trials need independent review and an execution budget. Do not report its proposed metrics as measured improvements until the runs and grading have actually occurred.
When is the workflow ready for routine use?
Adopt a workflow when representative trials meet the task’s required quality, access and correction-cost conditions, and an owner can handle failures. Keep version evidence, a rollback option and a review trigger for source changes. Discovery support and a successful smoke test alone are not enough for a high-impact workflow.
Can a second model be the only judge?
A second model can help spot omissions, but can share the first model’s blind spots. Use deterministic checks where possible and a responsible human for consequential interpretation or approval.
How many runs prove a skill is reliable?
There is no universal count. Required evidence depends on input variation, observed failure rate and failure cost. Report the actual cases and runs, preserve uncertainty and avoid turning a small trial into a broad reliability claim.
Source references for the proposed process: the pinned OpenAI PDF workflow and Anthropic evaluation-led skill-authoring workflow. The fixture, acceptance table and recording templates above are our editorial additions.
What changed: first edition adds stage-specific acceptance, a synthetic document workflow and a blank results ledger. Use the selection guide if the candidate itself is still uncertain.