18KB
Skill Extractor
Version: 1.0.0 Status: ACTIVE
Purpose
Turn reusable knowledge discovered during real project work into a tested, maintainable skill.
This skill is the main bridge between:
EXPERIENCE
and:
REUSABLE AGENT CAPABILITY
It allows the agent system to improve itself based on actual work instead of guessing in advance what skills might be useful.
1. Trigger Conditions
Use this skill after substantial work when one or more of these occur:
- the same procedure is likely to recur
- a repeated failure was finally solved
- the same manual workflow appeared more than once
- the user provided a durable working method
- a successful technique clearly saved time or reduced errors
- multiple tasks share the same operational pattern
- a new validation method repeatedly catches problems
- the same sequence of discovery steps keeps being used
- a recurring project convention needs to be followed consistently
Do not create a skill merely because a task was complicated.
Complexity alone does not imply reuse.
2. Core Principle
Use:
REAL WORK
↓
OBSERVE PATTERN
↓
GENERALIZE
↓
TEST
↓
SKILL
not:
IMAGINE POSSIBLE FUTURE TASK
↓
CREATE SKILL
Skills should grow organically from actual project needs.
3. First Question
After meaningful work, ask:
Did this task reveal a reusable:
procedure?
decision rule?
failure pattern?
verification method?
diagnostic technique?
tool workflow?
review process?
automation opportunity?
If the answer is no:
DO NOT CREATE A SKILL
4. Capture the Observation
Before creating anything permanent, record the candidate in:
.ai/state/LEARNINGS.md
Capture:
TASK CONTEXT
WHAT WORKED OR FAILED
EVIDENCE
POSSIBLE REUSABLE LESSON
CONFIDENCE
This prevents weak observations from becoming permanent skills too quickly.
5. Separate the Layers
Break the observed process into three categories.
STABLE PROCEDURE
PROJECT-SPECIFIC VARIABLES
ONE-OFF DETAILS
Example:
Observed deployment:
stop ScheduliciousPool
backup C:\Sites\Schedulicious
copy files
restart pool
test /health
Stable procedure:
identify application runtime
create rollback point
stop or drain application
deploy
restart
verify health
rollback if verification fails
Project variables:
application pool name
deployment directory
health-check URL
One-off details:
specific failure from today's deployment
The skill should preserve the stable procedure and parameterize the project variables.
6. Search Existing Skills
Before creating a new skill:
SEARCH .ai/skills/
Look for:
- same trigger
- same procedure
- overlapping responsibility
- a more general skill that can be extended
Prefer:
IMPROVE EXISTING SKILL
over:
CREATE NEAR-DUPLICATE SKILL
7. Skill Ownership Test
Ask:
Does an existing skill logically own this behavior?
Example:
Observed lesson:
Production database migrations should verify row counts after transformation.
If there is already:
.ai/skills/database-migration/SKILL.md
update it.
Do not create:
.ai/skills/database-row-count-check/SKILL.md
unless that procedure truly stands alone across many different workflows.
8. New Skill Justification
Create a new skill only when all are reasonably true:
REUSABLE
RECOGNIZABLE TRIGGER
MEANINGFUL PROCEDURE
CLEAR OUTPUT
VERIFIABLE
NOT ALREADY OWNED ELSEWHERE
The ideal model is:
INPUT
↓
PROCEDURE
↓
OUTPUT
↓
VERIFICATION
9. Skill Naming
Use a name that describes the job.
Good:
access-csv-import
iis-deployment
database-migration
api-security-review
classic-asp-controller
maildat-certified-mail
Weak:
helper
useful-stuff
project-skill
advanced-agent
skill-27
Use:
lowercase-kebab-case
for directory names.
10. Create from Template
Use:
.ai/templates/SKILL_TEMPLATE.md
or:
.ai/skills/skill-builder/SKILL.md
Every substantial skill should contain:
PURPOSE
TRIGGER
INPUTS
PRECONDITIONS
PROCEDURE
OUTPUT CONTRACT
VERIFICATION
FAILURE HANDLING
EXAMPLES when useful
SELF-IMPROVEMENT RULES
VERSION
CHANGELOG
11. Define the Trigger Carefully
A skill should be easy for another agent to know when to load.
Weak trigger:
Use when working with databases.
Better:
Use when importing delimited files into a Microsoft Access database and the import must validate schema, errors, duplicates, and row counts.
Precise triggers reduce unnecessary skill loading.
12. Generalize Without Losing Value
Avoid over-generalizing.
Weak extracted skill:
1. Understand the task.
2. Do the task.
3. Check the task.
That adds no reusable knowledge.
Preserve the important operational details.
13. Do Not Overfit
Avoid preserving accidental specifics.
Bad:
Always import Territory223.csv into Table223.
Better:
Validate the source file against the destination schema before import.
Project-specific constants can become parameters.
14. Preserve Real Domain Knowledge
Do not remove domain-specific rules that make the procedure useful.
Example:
If a certified-mail Mail.dat process genuinely requires a relationship between specific file types, keep that domain rule.
Generalization should remove accidental details, not essential domain knowledge.
15. Extract Decisions
Look for decision points.
Example:
IF source schema matches
continue
IF schema differs
stop and map fields
IF duplicate key exists
use defined duplicate policy
Decision rules are often the most reusable part of a skill.
16. Extract Preconditions
Ask:
What had to be true before the procedure could safely begin?
Examples:
backup exists
service account has permission
destination table exists
source file is complete
deployment package passed build
Capture important preconditions explicitly.
17. Extract Verification
Ask:
What proved that the procedure succeeded?
Examples:
build passed
row counts match
health check returned success
expected files exist
test record is correct
service is running
API returns expected response
Verification should be first-class skill logic.
18. Extract Failure Handling
Ask:
What happened when the normal path failed?
Capture repeated high-value failure behavior.
Examples:
rollback deployment
quarantine bad input
retry idempotently
stop migration
restore backup
produce rejection report
19. Extract Safety Boundaries
For consequential skills identify:
WHAT MUST NEVER HAPPEN?
Examples:
never delete source before import verification
never overwrite production config without backup
never continue migration after validation failure
never log credentials
Safety boundaries are part of the reusable procedure.
20. Identify Parameters
Convert project-specific details into explicit inputs.
Example:
APPLICATION_POOL
DEPLOYMENT_PATH
HEALTH_URL
BACKUP_PATH
This makes the skill reusable without stripping away useful structure.
21. Test on the Original Scenario
The first test should reproduce the original task.
Ask:
If this skill had existed before the task,
would it have led us to the successful process?
If not, the extraction missed something.
22. Test on a Second Scenario
A skill is not proven reusable until it works somewhere different.
Example:
Original:
Deploy ASP Classic application.
Second test:
Deploy ASP.NET application using the same IIS workflow.
If the skill breaks because too many details were hard-coded, revise it.
23. Transfer Test
Check:
Can different input values be substituted?
Does the procedure still make sense?
Does the trigger remain clear?
Do verification rules transfer?
Do failure-handling rules transfer?
If yes, the skill is likely well generalized.
24. Counterexample Test
Try one scenario where the skill should NOT apply.
Example:
Skill:
database-migration
Counterexample:
simple SELECT query optimization
If the skill's trigger would incorrectly activate, tighten the trigger.
25. Skill Quality Test
A good skill should reduce at least one of:
rediscovery
clarification
manual steps
errors
prompt retries
verification gaps
inconsistent implementation
If it does none of these, it may not deserve to exist.
26. Create Supporting Assets
A skill may justify supporting files.
Examples:
SKILL.md
CHECKLIST.md
template.ps1
validation.sql
example-config.json
Only create supporting files when they improve execution.
Do not create files merely to make the skill look complete.
27. Skills May Create Other Skills
A skill may reveal subprocedures that deserve their own skills.
Example:
iis-deployment
repeatedly uses:
certificate-validation
If certificate validation becomes a substantial reusable procedure across other tasks, it may become its own skill.
Before splitting, ask:
Is this subprocedure independently reusable?
28. Skills May Create Prompts
If part of the procedure is a repeated AI interaction, create a supporting prompt.
Example:
database-migration/SKILL.md
may use:
.ai/prompts/migration-risk-review.md
The skill owns the procedure.
The prompt launches a specific interaction.
29. Skills May Create Templates
Example:
A postmortem skill may repeatedly produce the same document shape.
It may create:
.ai/templates/POSTMORTEM_TEMPLATE.md
Avoid embedding large templates directly into every skill if a reusable template is cleaner.
30. Skills May Update AGENTS.md
If a newly created skill becomes important enough that most agents need to know when to invoke it, add a SHORT routing rule to:
AGENTS.md
Example:
For production database schema changes, use the database-migration skill.
Do not copy the entire migration procedure into AGENTS.md.
31. Skills May Update CLAUDE.md
Only when the lesson concerns Claude-specific execution.
Example:
Claude should use repository search to locate existing deployment scripts before loading the deployment skill.
General skill behavior belongs elsewhere.
32. Skill Creation Evidence
The skill's changelog should identify its origin.
Example:
Evidence:
This procedure appeared in four deployments and prevented
partial-copy failures in the two most recent deployments.
This allows future agents to understand why the skill exists.
33. Version the Skill
Start meaningful active skills at:
1.0.0
Candidate skills may start:
0.1.0
Use PATCH, MINOR, and MAJOR according to:
.ai/templates/SKILL_TEMPLATE.md
34. Update the AI Changelog
When creating or materially changing a reusable skill, update:
.ai/CHANGELOG.md
Record:
skill created or changed
reason
evidence
expected improvement
verification
35. Update the Learning Ledger
After successful promotion, change the original learning entry to:
Action: PROMOTED
Record:
Candidate Destination:
.ai/skills/...
Result After Reuse:
...
when future evidence becomes available.
36. Avoid Skill Explosion
A self-improving system can become worse if it creates hundreds of tiny skills.
Before creating a skill ask:
Will another agent realistically search for and use this?
If no, do not create it.
Prefer a smaller set of high-value skills.
37. Merge Similar Skills
Suppose the system contains:
csv-import
excel-import
text-file-import
and all use nearly identical:
schema validation
staging
error capture
row-count checks
Consider merging into:
structured-data-import
with format-specific branches.
Merge only if that improves clarity.
38. Split Oversized Skills
A skill may become too large when:
trigger conditions are unrelated
procedure has many independent branches
different parts are reused independently
agents load a lot of irrelevant guidance
Then split.
Example:
windows-server-management
might eventually split into:
iis-deployment
windows-firewall
openssh-setup
certificate-management
if actual usage justifies it.
39. Skill Dependency Rules
A skill may reference other skills.
Example:
deployment skill
↓
mirror-audit
↓
red-team
Do not copy their complete instructions.
Reference them.
This keeps skills modular.
40. Skill Dependency Loops
Avoid circular dependencies such as:
Skill A requires Skill B
Skill B requires Skill A
If this occurs, identify the shared procedure and move it into:
a common skill
or:
AGENTS.md
if truly universal.
41. Skill Conflict Detection
When creating a new skill, compare its rules with:
AGENTS.md
CLAUDE.md
WORKFLOW.md
related skills
Resolve conflicts before activation.
Run:
.ai/skills/system-audit/SKILL.md
42. Skill Activation
A skill becomes ACTIVE when:
trigger is clear
procedure is documented
verification exists
original scenario passes
transfer scenario passes
system audit passes
Before that it may remain:
CANDIDATE
or:
DRAFT
43. Deprecation
A skill should be deprecated when:
technology disappears
workflow changes
another skill replaces it
project facts make it obsolete
tests repeatedly show worse outcomes
Mark it clearly or remove it according to project policy.
Do not leave obsolete active instructions.
44. Deprecation Learning
When a skill is deprecated, ask:
Why did it stop being useful?
The answer may itself be useful learning.
Example:
Skill assumed single-server deployment.
Project moved to containers.
That tells the AI-OS which assumptions have changed.
45. Example — PowerShell Deployment Skill
Observed pattern:
verify package
backup current release
stop app pool
deploy
start app pool
test endpoint
rollback if failure
Repeated three times.
Candidate:
.ai/skills/iis-deployment/SKILL.md
Inputs:
APP_POOL
SOURCE_PATH
DESTINATION_PATH
HEALTH_CHECK
Verification:
app pool running
health endpoint successful
expected version deployed
This is a strong skill candidate.
46. Example — Weak Skill Candidate
Observation:
PowerShell command needed Get-Date once.
Candidate skill:
get-current-date
Reject.
Reason:
trivial
not a meaningful workflow
already obvious from general tooling
47. Example — User Correction
Repeated user correction:
Do not use React in this application.
Use server-rendered HTML and existing JavaScript.
This is probably not a skill.
Likely destination:
project documentation
or a short project rule.
Skill extraction includes knowing when NOT to create a skill.
48. Example — Troubleshooting Skill
Across several incidents, the successful method is:
capture exact error
identify network/application/database layer
test connectivity
test authentication
test application call
compare working vs failing host
This may justify:
.ai/skills/layered-troubleshooting/SKILL.md
because the procedure generalizes.
49. Example — Verification Skill Update
Existing deployment skill says:
verify service is running
A failure occurs where service is running but application is broken.
Learning:
service state alone is insufficient.
Improve existing skill:
verify service state
+
application health
Do not create a separate:
application-health-after-deployment
skill unless independently reusable.
50. Skill Extraction Decision Tree
REUSABLE LESSON?
|
+--+--+
| |
NO YES
| |
DONE v
EXISTING OWNER?
/ \
YES NO
| |
UPDATE v
SUBSTANTIAL?
/ \
NO YES
| |
ROOT/DOC NEW SKILL
|
v
TEST
|
v
SYSTEM AUDIT
|
v
LOG
51. Self-Improvement
Improve this skill when:
- too many low-value skills are being created
- useful patterns are not being captured
- skills are overfitting original cases
- overlap between skills increases
- testing does not catch poor generalization
- the system needs better merge/split criteria
The extractor itself should improve based on the quality of the skill library it produces.
52. Output Contract
When this skill is invoked, produce an internal or explicit decision containing:
REUSABLE LESSON
EVIDENCE
EXISTING OWNER SEARCH
DESTINATION DECISION
GENERALIZED PROCEDURE
TEST CASES
PROMOTION DECISION
If creating a new skill, produce the completed skill file.
If updating an existing skill, make the smallest justified revision.
If no skill is justified:
NO SKILL CREATED
is a successful outcome.
53. Changelog
1.0.0
Initial active version.
Introduced:
- experience-driven skill creation
- learning-ledger staging
- stable/project/one-off separation
- existing-skill search
- skill ownership test
- trigger design
- generalization and overfitting checks
- original and transfer testing
- skill creation/merge/split rules
- supporting prompts/templates
- skill routing into AGENTS.md
- activation and deprecation lifecycle