Home / Methodology

How SkillCheck checks a skill

Every check traces to a published spec, a research paper or documented practitioner experience, not to opinion. When we add one, we document what informed it and why it matters.

Built on: Anthropic guidance · W3C WCAG 2.2 · OWASP Top 10 for Agentic Applications · OpenAI · Google Cloud · the MCP specification · Hasan et al. and Wang et al. (arXiv) · all 22 sources

Scores and grades

Every skill starts at 100 points. Each finding takes points off, depending on how serious it is.

FindingPointsWhat it means
Critical−20Breaks the skill or makes it unsafe. Fix before shipping.
Warning−5A quality gap an agent or a user will notice. Should fix.
Suggestion−1Worth improving. Suggestions cost at most 15 points in total, so a long, thorough skill is not punished for ordinary wording.
Strength0Something done well, such as a strong gotchas section. Shown in the report; it does not change the score.

From score to grade

GradeScoreWhat it means
Excellent90–100Clear and safe to depend on. Only small suggestions left.
Good75–89Works well. A few warnings worth fixing.
Needs Work60–74Usable, but with problems an agent or a user will notice.
Poor0–59Fix before sharing.

A Critical finding caps the grade at Needs Work. One Critical costs 20 points, so an otherwise clean skill would still score 80. The cap stops a report from calling a skill Good while it lists a problem that must be fixed first. The same four grades appear in the command line, in reports and on this site.

How a check works

Most checks match published patterns line by line, so the same skill always gets the same result.

Structural

Present or absent

Required fields, file references, secrets, token counts. Exact: pass or fail.

Structure · Body · Security · Token budget · Trigger collision

Pattern

Named patterns

Filler phrases, density signals, design patterns, governance checklists. Read the rules and you can predict the result.

Naming · Anti-slop · Enterprise · OWASP deterministic · Knowledge density

Judgment

Reading comprehension

Contradictions, workflow clarity, autonomy boundaries. Scored against a published rubric, so the tolerance is wider.

Semantics · Workflow · Autonomy design · OWASP graders

What we hold the checks to

Standards, not opinions

Every check traces to a published spec, a research paper or documented practitioner experience. We record what informed it and why it matters.

Layers, in order

Structure first, then meaning, then content quality, security and agent readiness.

Reward what is good

A strong gotchas section or clear error handling shows up in the report as a strength, not only as missing penalties.

Where the checks come from

SkillCheck grew in eleven phases. Each one added checks grounded in a new, independent source. Open a phase to see what informed it.

01Build to specv1.0 – v1.4 · Dec 2025Categories 1–10

Foundation checks built directly from Anthropic's official guidance: frontmatter fields, trigger conditions, scope boundaries, testing scenarios. WCAG 2.2 informed accessibility; MCP Best Practices established protocol compliance.

Categories 1–10: Structure · Body · Naming · Semantics · Anti-Slop · Visual · Security · Quality · Token · Enterprise

  • Anthropic Claude Code Best PracticesFrontmatter schema, trigger conditions, scope boundaries, testing scenarios
  • MCP Protocol MCP Best PracticesProtocol compliance, tool-list format, server registration
  • W3C WCAG 2.2Colour contrast thresholds, alt-text guidance, accessibility output checks
02Build from practicev3.10 – v3.14 · Mar 2026Categories 11–18

Practitioner insights from shipping skills at scale. Field observations from the Claude Code team introduced description-as-trigger-condition, railroading detection, and gotchas as a quality signal. Agent Readiness added 6 pillars and 28 checks for multi-agent orchestration.

Categories 11–18: Quality Pro · Workflow · Reference Integrity · Eval Readiness · Orchestration Safety · Autonomy Design · Composability · Observability

03Build from the fieldv3.16 – v3.17 · Apr 2026Categories 19–21

Cross-lab methodology. OpenAI's systematic eval approach informed the Eval Kit. Google's ADK agent-type hierarchy inspired SkillCheck's design-pattern classification. CI/CD integration brought SkillCheck into GitHub Actions.

Categories 19–21: Design Pattern · Trigger Collision · Eval Kit

04Build from the ecosystemv3.18.0 · Apr 2026Category 22

Academic research and community tools. Hasan et al. found 97% of 856 MCP tools contained a description smell. Wang et al.'s 18-category taxonomy showed standard-compliant descriptions hit 72% selection probability vs a 20% baseline.

Category 22: Knowledge Density

05Build for the marketplacev3.20.0 · Apr 2026Categories 23–25

Scope widens from SKILL.md to three artifact types. Cat 23 evaluates MCP servers on four axes. Cat 24 checks plugin manifests against the Anthropic schema. A 21-plugin benchmark confirmed calibration: zero criticals, but 0/21 declare maintainers or evals.

Categories 23–25: Agent Integration Readiness · Skill Marketplace Governance · Memory Governance (design)

06Build for agentic safetyv3.21.0 · Jun 2026Category 26

Cat 26 checks a skill against the OWASP Top 10 for Agentic Applications (2026). Eight deterministic risks run free and local, in CI and the browser. Three intent-based risks are Pro graders your own AI judges, with no API key required.

Category 26: OWASP Agentic Top 10: tool over-grant · identity override · unpinned deps · code execution · unbounded loops · ungated destructive actions · disabled safeguards · no audit trail · goal hijacking (Pro) · memory poisoning (Pro) · insecure inter-agent comms (Pro)

07Measure lived-in-ness, not just craftv3.22.0 · Jun 2026Category 27

Cat 27 reads a skill's git history to answer what the SKILL.md text can't: is anyone actually using this, or was it written once and abandoned? Commit count, recency, and commit-day spread flag skills written once and never revised. It scores in its own domain, so a one-commit skill can still be excellent.

Category 27: Repo Maturity: commit count · recency · single-sitting · actively-iterated (strength). Pro, git-history.

  • Vanishing Gradients Show Us Your (Agent) SkillsSkepticism toward one-off skills: many published skills carry a single commit, never iterated by an author who depends on them
08Read the whole skill, reward restraintv3.24 – v3.26 · Jun–Jul 2026Category 28

Cat 28 rewards skills that say 'keep it simple' AND 'never skip validation, security, accessibility'; lazy is fine, careless is not. Reference-aware checking teaches content checks to follow a skill's references/ files instead of flagging a lean SKILL.md as incomplete: first eval-readiness (v3.25), then contracts, confidence signals, and knowledge density (v3.26). Calibrated against the July 2026 corpus rescan: 2,675 public skills, 1 in 5 shipping references/, 2 in 5 among the most popular.

Category 28: Restraint · reference-aware expansion of Categories 14, 17, 18, 22

09Close the slop-coverage gapsv3.27 – v3.28 · Jul 2026Category 5 deepened

No new category: two rounds of deepening Category 5 against external catalogues of AI writing rather than against taste. v3.27 ported cluster-mined patterns: an AI-vocabulary list where a single word family stays a Suggestion but two co-occurring families escalate every hit, plus significance inflation, cross-sentence antithesis, and corporate/internet cliché lists. v3.28 audited coverage against Wikipedia's catalogue, found nine applicable categories with no pattern behind them, and closed four. One of them, chat-UI citation markup pasted into a skill, is the first Category 5 pattern scored Critical rather than Warning: every other pattern has a legitimate use, and this one is verbatim paste evidence. Five categories are deliberately left uncovered and written down as decisions, so a later reader can tell a choice from an oversight.

Category 5 deepened: chat-citation-artifact (Critical) · knowledge-cutoff · collab-communication · copula-inflation · ai-vocabulary

  • Wikipedia Signs of AI writingCoverage audit target; the nine applicable categories and the four closed in v3.28, including the chat-citation artifact family
  • Adam Dunkels deslop-textCluster-mined vocabulary and cliché families ported into 5.3 and 5.4
10Build for the mountv3.31.0 · Aug 2026Category 29

Cat 29 asks a repository-level question the SKILL.md text cannot answer: will these skills actually be discovered when the repository is mounted into an Anthropic Managed Agents session? The platform loads skills from exactly .claude/skills/<name>/SKILL.md, one level deep at the root, with no review step, and a skill placed one directory too deep never errors or logs, it simply never loads. Anthropic's documentation tells users to audit that directory before mounting a repository that accepts outside contributions, without naming a tool. Cat 29 is that audit: the four silent-load layouts, empty skill directories, symlinks escaping the repo, name mismatches and case collisions run free; a tamper-evident evidence manifest with a SHA-256 per mountable skill is the Pro part, so CI fails on layout regressions and content drift.

Category 29: Skill Distribution Readiness: silent-load layouts · empty skill dirs · escaping symlinks · name mismatches · case collisions · mountable-skill threshold · evidence manifest (Pro)

11See what the reviewer cannotv3.32.0 · Aug 2026Category 30

Cat 30 came out of reading a competitor. NVIDIA's SkillSpector is a security scanner for agent skills; most of it scans code, outside SkillCheck's scope, but it exposed one gap that was in scope: nothing in SkillCheck detected instructions hidden in invisible characters. A SKILL.md can carry an entire instruction in Unicode tag characters that no editor and no diff will show and every model will read. Four checks now run free and in the browser: ASCII smuggling in tag characters, with the hidden payload decoded into the finding so the reviewer sees exactly what was smuggled; trojan-source bidirectional overrides; long invisible runs as an encoding channel; and stray zero-width characters. Legitimate emoji are exempt by construction, because a flag emoji is built from the same tag block as the attack.

Category 30: Invisible Unicode: ascii-smuggling (Critical, payload decoded) · bidi-override (Critical) · invisible-run · invisible-char

  • NVIDIA SkillEvaluatorInvisible-codepoint categories and run-length thresholds (Apache-2.0)
  • NVIDIA SkillSpectorThe in-scope gap that started the category, and the emoji tag-sequence carve-out
  • Boucher & Anderson Trojan Source (CVE-2021-42574)Bidirectional-override attacks where displayed text differs from parsed text

All sources

Every source that informed a check, with what it informed and the phase it belongs to.

SourceWhat it informedPhase
Anthropic
Claude Code Best Practices
Frontmatter schema, trigger conditions, scope boundaries, testing scenarios01
MCP Protocol
MCP Best Practices
Protocol compliance, tool-list format, server registration01
W3C
WCAG 2.2
Colour contrast thresholds, alt-text guidance, accessibility output checks01
Anthropic
Lessons from Building Claude Code (Thariq Shihipar)
Description-as-trigger condition, railroading detection, gotchas as quality signal, Agent Readiness pillars02
OpenAI
Testing Agent Skills Systematically with Evals
Should-trigger / should-NOT-trigger prompt pairs, negative controls per validated skill03
Google
5 Agent Skill Design Patterns (Google Cloud Tech)
Reviewer / Generator / Inversion / Pipeline / Tool Wrapper classification03
Hasan et al. · arXiv
MCP Tool Descriptions Are Smelly!
Description smell taxonomy, 97% prevalence in 856 public MCP tools04
Wang et al. · arXiv
From Docs to Descriptions
18-category taxonomy; compliant descriptions → 72% selection vs 20% baseline04
Glama
Tool Description Quality Score (TDQS)
Community-scale calibration of description quality signals04
Anthropic
Building Agents That Reach Production with MCP
MCP server governance axes: token efficiency, security, unique unlocks, execution environment05
Sam Morrow
MCP vs CLIs
CLI literacy checks, when a tool wraps a CLI vs implements its own protocol05
Anthropic
anthropics/knowledge-work-plugins
Reference schema for plugin manifests; benchmark of 21 real plugins05
OWASP
Top 10 for Agentic Applications 2026
All 11 risk categories: 8 mapped to deterministic checks, 3 to AI-judge graders06
Vanishing Gradients
Show Us Your (Agent) Skills
Skepticism toward one-off skills: many published skills carry a single commit, never iterated by an author who depends on them07
Dietrich Gebert
ponytail ('lazy senior dev' skill)
Tests showed 'do less' instructions cut code but sometimes dropped a safety guard; restraint needs a carveout08
SkillCheck
Full-corpus rescan, July 2026 (117 published reports)
references/ adoption at 21% corpus-wide vs 41% among featured skills; validators that stop at SKILL.md tax the converging layout08
Wikipedia
Signs of AI writing
Coverage audit target; the nine applicable categories and the four closed in v3.28, including the chat-citation artifact family09
Adam Dunkels
deslop-text
Cluster-mined vocabulary and cliché families ported into 5.3 and 5.409
Anthropic
Managed Agents: Skills (documentation, snapshot 10-08-2026)
The discovery contract: exact path, one level deep, scan once at session start, read tool required, audit before mounting10
NVIDIA
SkillEvaluator
Invisible-codepoint categories and run-length thresholds (Apache-2.0)11
NVIDIA
SkillSpector
The in-scope gap that started the category, and the emoji tag-sequence carve-out11
Boucher & Anderson
Trojan Source (CVE-2021-42574)
Bidirectional-override attacks where displayed text differs from parsed text11

Independence

SkillCheck is an independent project, not affiliated with or endorsed by Anthropic, OpenAI, Google or any other AI lab.

Research from those organizations informed specific check categories, as listed in the phases above. The implementation, the scoring and the quality judgments are SkillCheck's own.

Run the checks on your own skill.

The free check runs in your browser. Pro adds security, filler text, accessibility and agent readiness.