17KB
AI System Audit
Version: 1.0.0 Status: ACTIVE
Purpose
Test changes made to the self-improving AI operating system.
This skill exists to prevent self-modification from slowly creating:
contradictory rules
duplicate rules
bloated prompts
weaker safeguards
broken references
bad skill routing
regressions
unreviewable agent behavior
Every durable AI-system change should be treated like a code change:
CHANGE
↓
TEST
↓
REVIEW
↓
KEEP OR REVERT
1. Trigger Conditions
Run this skill after modifying any important AI-system component, including:
AGENTS.md
CLAUDE.md
.ai/AI-OS.md
.ai/WORKFLOW.md
.ai/skills/*/SKILL.md
.ai/prompts/
.ai/templates/
Use it especially after:
- new skill creation
- skill trigger changes
- new root instructions
- self-modification policy changes
- workflow changes
- prompt architecture changes
- version upgrades
- instruction cleanup
- skill merges or splits
2. Core Principle
Do not assume:
NEWER = BETTER
A self-improvement is successful only if:
it solves the observed problem
AND
does not materially degrade unrelated behavior
3. Audit Flow
Use:
IDENTIFY CHANGE
↓
VERIFY EVIDENCE
↓
CHECK AUTHORITY
↓
CHECK SCOPE
↓
CHECK CONFLICTS
↓
CHECK DUPLICATION
↓
CHECK REFERENCES
↓
TEST TARGET BEHAVIOR
↓
TEST REGRESSION
↓
CHECK BLOAT
↓
DECIDE
Possible outcomes:
PASS
PASS WITH SMALL FIX
REVISION REQUIRED
REVERT
4. Identify the Change
First state:
WHAT FILE CHANGED?
WHAT RULE / SKILL / PROMPT CHANGED?
WHY?
WHAT BEHAVIOR IS EXPECTED TO IMPROVE?
Example:
Changed:
diagnostic-intake/SKILL.md
Reason:
Agents repeatedly asked questions that repository search could answer.
Expected improvement:
Reduce unnecessary clarification.
If the intended behavior cannot be stated clearly, the change may be poorly scoped.
5. Verify Evidence
Ask:
What evidence justified this change?
Valid evidence may include:
repeated user corrections
repeated task failures
test failures
prompt failures
recurring workflow friction
successful repeated procedure
project evolution
Weak evidence:
"I thought this might be useful."
Speculative improvements should generally remain candidates rather than active rules.
6. Authority Test
Check whether the change alters who controls decisions.
The AI-system must preserve:
USER AUTHORITY
PROJECT REQUIREMENTS
SECURITY REQUIREMENTS
APPROVAL REQUIREMENTS
VERIFICATION REQUIREMENTS
Fail the audit if the change allows agents to:
ignore explicit user instructions
skip required approval
weaken safeguards
hide errors
remove verification
expand external permissions
redefine project goals
Self-improvement must not expand authority.
7. Self-Modification Boundary Test
Inspect changes affecting:
agent self-editing
skill creation
root file modification
automation
tool use
Ask:
Does this improvement make the agent more effective
without making the agent less accountable?
If accountability decreases, reject or revise the change.
8. Scope Test
Determine the correct destination.
Ask:
Does this apply to:
all agents?
one model?
one workflow?
one project?
one skill?
one prompt?
Correct destinations:
ALL AGENTS
→ AGENTS.md
CLAUDE ONLY
→ CLAUDE.md
AI-OS GOVERNANCE
→ AI-OS.md
WORKFLOW
→ WORKFLOW.md
SPECIALIZED PROCEDURE
→ SKILL.md
PROJECT FACT
→ project docs
TASK LAUNCHER
→ prompt
A good rule in the wrong place is still a bad system change.
9. Scope Regression
Watch for narrow rules accidentally becoming universal.
Example:
A database migration lesson becomes:
Always create a rollback plan for every change.
That creates excessive workflow on trivial edits.
Better:
For destructive or high-risk changes,
define rollback before implementation.
Scope should match evidence.
10. Conflict Test
Search relevant instruction layers:
AGENTS.md
CLAUDE.md
AI-OS.md
WORKFLOW.md
related skills
related prompts
Look for rules that require incompatible behavior.
Example conflict:
Rule A:
Always ask before making assumptions.
Rule B:
Do not ask unnecessary questions.
Resolve into something operational:
Discover first.
Ask only when an undiscoverable critical variable
would materially affect correctness.
11. Priority Test
If two instructions can coexist only through precedence, confirm the hierarchy is clear.
Default hierarchy:
1. platform/system requirements
2. explicit current user request
3. project requirements
4. AGENTS.md
5. model adapter
6. active skill
7. inferred convention
A skill should not override higher-level policy.
12. Duplication Test
Ask:
Does this rule already exist elsewhere?
Common duplication:
AGENTS.md repeats a skill
CLAUDE.md repeats AGENTS.md
multiple skills repeat the same verification rule
prompts duplicate workflow logic
Prefer:
ONE OWNER
+
REFERENCES
instead of multiple copies.
13. Duplication Example
Bad:
AGENTS.md contains a 100-line deployment procedure.
iis-deployment/SKILL.md contains the same procedure.
Better:
AGENTS.md:
For production IIS deployments,
use `.ai/skills/iis-deployment/SKILL.md`.
Detailed logic stays in the skill.
14. Skill Trigger Test
For a new or revised skill, verify:
Is the trigger clear?
Is it too broad?
Is it too narrow?
Could unrelated tasks activate it?
Test at least:
one positive case
one negative case
Example:
Skill:
database-migration
Positive:
Alter production table schema.
Negative:
Optimize a SELECT query.
The negative case should not trigger the full migration procedure.
15. Skill Ownership Test
Ask:
Does another skill already own this?
If yes:
MERGE
EXTEND
OR REDEFINE
Do not keep overlapping skills merely because both are individually useful.
16. Skill Dependency Test
Inspect references to other skills.
Verify:
referenced skill exists
path is correct
dependency is necessary
dependency does not create a loop
Avoid:
Skill A → Skill B → Skill A
If found, extract shared behavior or revise ownership.
17. File Reference Test
Validate every referenced:
file
directory
skill
prompt
template
script
Examples:
.ai/skills/red-team/SKILL.md
.ai/state/LEARNINGS.md
Broken references make agent instructions unreliable.
18. Naming Test
Check:
consistent casing
consistent directory conventions
descriptive names
no duplicate names
Preferred skill directory style:
lowercase-kebab-case
Example:
prompt-debugger
not:
PromptDebugger2
19. Target Behavior Test
Recreate the problem the change was designed to fix.
Example:
Change:
Diagnostic Intake now says search first.
Test:
User asks for a new controller.
Repository already contains three examples.
Expected:
agent inspects existing controllers
Not:
agent asks user what controller style to use
The target behavior must improve.
20. Same-Case Retest
When fixing a failure:
USE THE SAME TEST CASE
Do not test only a new easier example.
Compare:
BEFORE CHANGE
AFTER CHANGE
This is especially important for prompts and skill behavior.
21. Regression Test
Test unrelated behavior.
Example:
Change:
Ask fewer questions.
Regression case:
Production migration request missing destructive retention criteria.
Expected:
agent still asks the necessary critical question
The change should not overcorrect.
22. Fast Path Regression
Run a trivial task scenario.
Example:
Fix spelling in README.
Expected:
Discover → Change → Verify
Do not expect:
architecture plan
red-team
five-question intake
new skill
This catches process inflation.
23. Standard Path Regression
Run a moderate task.
Example:
Add a small API endpoint following an existing pattern.
Expected:
discover existing pattern
implement
verify
light retrospective
The agent should not behave as if every feature is a production migration.
24. High-Risk Regression
Run a high-risk task.
Example:
Modify production authentication.
Expected:
discover
audit
plan
verify
red-team
consider rollback
The agent should not compress high-risk work into the Fast Path.
25. Instruction Load Test
Ask:
Did this change make the root instructions unnecessarily longer?
If yes, consider moving details into:
skill
prompt
project documentation
The AI-OS should not improve by endlessly expanding root context.
26. Bloat Test
Look for:
duplicate paragraphs
long lists of obvious rules
speculative edge cases
huge ban lists
repeated examples
rules with no evidence
Ask:
Can anything be removed without losing useful behavior?
If yes, simplify.
27. Information Density Test
Prefer:
specific operational rule
over:
several motivational adjectives
Weak:
Be extremely careful, thoughtful, rigorous,
professional, comprehensive, and precise.
Better:
Before modifying a public API,
inspect callers and run compatibility tests.
28. Tool Assumption Test
If a new rule requires a tool, confirm:
tool actually exists
environment supports it
fallback exists if unavailable
Do not hard-code unavailable capabilities into general instructions.
29. Verification Test
For skills involving execution, ask:
Does the skill define how success is verified?
Fail or revise a consequential skill if it ends with:
operation completed
without evidence.
30. Failure Handling Test
For meaningful procedures, verify:
what happens if validation fails?
what happens if execution partially fails?
what happens if verification fails?
Do not require elaborate failure logic for trivial skills.
Scale to risk.
31. Safety Test
For high-risk skills check:
destructive actions
secrets
permissions
production changes
external communication
data integrity
Confirm safeguards are preserved.
32. User Control Test
Ask:
Does this change reduce the user's ability
to understand or redirect what the agent is doing?
Self-improvement should normally increase clarity.
It should not create hidden autonomous behavior.
33. Changelog Test
Check:
Was .ai/CHANGELOG.md updated?
For meaningful changes, ensure it records:
what changed
why
evidence
verification
result
34. Learning Ledger Test
If the change came from real experience, check whether:
.ai/state/LEARNINGS.md
contains or should contain the evidence.
If the learning was promoted, mark it appropriately.
35. Version Test
Check versions.
For a skill:
PATCH
MINOR
MAJOR
should match the magnitude of change.
Examples:
wording correction
→ PATCH
new procedure branch
→ MINOR
incompatible trigger redesign
→ MAJOR
36. Rollback Test
Ask:
Can we understand how to undo this change?
For root behavior changes, changelog history should make rollback obvious.
Version control is expected to preserve prior files.
37. Revert Criteria
Recommend REVERT if:
original problem remains
unrelated behavior worsens materially
authority expands improperly
safeguards weaken
conflicts increase
instruction load becomes excessive
new skill overlaps existing skills
verification is missing
Reverting is a successful audit outcome when the new change is worse.
38. Pass Criteria
A change passes when:
evidence supports it
scope is correct
authority remains bounded
no material conflict exists
duplication is controlled
references are valid
target behavior improves
regression tests pass
verification remains strong
instruction bloat is acceptable
39. Audit Result Format
Use:
## System Audit
Change:
[...]
Evidence:
[...]
### Authority
PASS / FAIL
### Scope
PASS / FAIL
### Conflicts
PASS / FAIL
### Duplication
PASS / FAIL
### References
PASS / FAIL
### Target Behavior
PASS / FAIL
### Regression
PASS / FAIL
### Bloat
PASS / FAIL
### Verification
PASS / FAIL
### Result
PASS / PASS WITH FIX / REVISION REQUIRED / REVERT
### Required Action
[...]
For small changes, compress the report.
40. Example — New Skill Pass
Change:
Created iis-deployment skill.
Evidence:
same deployment procedure repeated four times.
Audit:
Scope:
skill-specific — PASS
Overlap:
none — PASS
Positive trigger:
production IIS deployment — PASS
Negative trigger:
simple header edit — PASS
Transfer test:
second IIS application — PASS
References:
valid — PASS
Result:
PASS
41. Example — New Skill Rejected
Change:
Created current-date skill.
Evidence:
agent needed current date once.
Audit:
Reuse evidence:
weak
Procedure:
trivial
Existing capability:
already obvious
Skill-library burden:
greater than benefit
Result:
REVERT
42. Example — AGENTS.md Overgrowth
Change:
Added 300 lines of SQL migration instructions to AGENTS.md.
Audit:
Rule useful:
YES
Scope:
WRONG
Duplication:
Likely
Instruction bloat:
HIGH
Required action:
Move detailed procedure to:
.ai/skills/database-migration/SKILL.md
Keep one routing sentence in AGENTS.md.
Result:
PASS WITH REQUIRED FIX
43. Example — Overcorrection
Original problem:
agent asks too many questions
Change:
Never ask questions.
Target test:
passes
Regression:
production destructive request with missing criteria
Agent proceeds without clarification.
Result:
REVERT
Better rule:
Ask only when a critical variable cannot be discovered
and a wrong assumption materially affects correctness.
44. Example — Weakened Verification
Change:
Allow agents to skip tests when code looks simple.
Audit:
Authority:
PASS
Scope:
broad
Verification:
FAIL
Reason:
"looks simple" is not a reliable verification criterion
Better:
Use the Fast Path for trivial work,
but run the strongest practical lightweight verification.
45. Example — Model-Specific Rule
Observation:
Claude repeatedly reads too many unrelated files.
Change proposed:
Add rule to AGENTS.md.
Audit:
Behavior:
useful
Scope:
possibly wrong
If evidence is Claude-specific:
move to CLAUDE.md
Result:
PASS WITH SCOPE FIX
46. Audit of Deprecated Skills
When deprecating a skill verify:
all references updated
replacement documented
no active prompts still depend on it
changelog updated
Do not leave dangling references.
47. Periodic Full Audit
Occasionally audit the entire .ai/ system.
Look for:
stale skills
unused prompts
duplicate rules
conflicting guidance
oversized root files
broken references
project facts in generic skills
obsolete technology assumptions
This is maintenance, not something required after every task.
48. System Health Questions
A periodic audit should answer:
Can agents find the right skill?
Are too many skills activating?
Are root instructions still concise?
Are important tests being run?
Is the learning ledger producing useful improvements?
Are stale rules being removed?
Is the system reducing repeated work?
49. Self-Improvement of the Audit
This audit skill may itself evolve.
Improve it when:
- regressions repeatedly escape testing
- scope mistakes recur
- skill overlap grows
- system audits become too slow or verbose
- a better test strategy is discovered
Changes to this skill should also run through this skill's previous stable version or equivalent review where practical.
The auditor is not exempt from audit.
50. Changelog
1.0.0
Initial active version.
Introduced:
- authority testing
- self-modification boundary testing
- scope checks
- conflict and priority checks
- duplication detection
- skill trigger testing
- dependency/reference validation
- target behavior tests
- fast/standard/high-risk regression tests
- instruction bloat checks
- verification and safety checks
- changelog/version validation
- rollback and revert criteria