# AI System Audit Version: 1.0.0 Status: ACTIVE ## Purpose Test changes made to the self-improving AI operating system. This skill exists to prevent self-modification from slowly creating: ```text contradictory rules duplicate rules bloated prompts weaker safeguards broken references bad skill routing regressions unreviewable agent behavior ``` Every durable AI-system change should be treated like a code change: ```text CHANGE ↓ TEST ↓ REVIEW ↓ KEEP OR REVERT ``` --- # 1. Trigger Conditions Run this skill after modifying any important AI-system component, including: ```text AGENTS.md CLAUDE.md .ai/AI-OS.md .ai/WORKFLOW.md .ai/skills/*/SKILL.md .ai/prompts/ .ai/templates/ ``` Use it especially after: * new skill creation * skill trigger changes * new root instructions * self-modification policy changes * workflow changes * prompt architecture changes * version upgrades * instruction cleanup * skill merges or splits --- # 2. Core Principle Do not assume: ```text NEWER = BETTER ``` A self-improvement is successful only if: ```text it solves the observed problem AND does not materially degrade unrelated behavior ``` --- # 3. Audit Flow Use: ```text IDENTIFY CHANGE ↓ VERIFY EVIDENCE ↓ CHECK AUTHORITY ↓ CHECK SCOPE ↓ CHECK CONFLICTS ↓ CHECK DUPLICATION ↓ CHECK REFERENCES ↓ TEST TARGET BEHAVIOR ↓ TEST REGRESSION ↓ CHECK BLOAT ↓ DECIDE ``` Possible outcomes: ```text PASS PASS WITH SMALL FIX REVISION REQUIRED REVERT ``` --- # 4. Identify the Change First state: ```text WHAT FILE CHANGED? WHAT RULE / SKILL / PROMPT CHANGED? WHY? WHAT BEHAVIOR IS EXPECTED TO IMPROVE? ``` Example: ```text Changed: diagnostic-intake/SKILL.md Reason: Agents repeatedly asked questions that repository search could answer. Expected improvement: Reduce unnecessary clarification. ``` If the intended behavior cannot be stated clearly, the change may be poorly scoped. --- # 5. Verify Evidence Ask: ```text What evidence justified this change? ``` Valid evidence may include: ```text repeated user corrections repeated task failures test failures prompt failures recurring workflow friction successful repeated procedure project evolution ``` Weak evidence: ```text "I thought this might be useful." ``` Speculative improvements should generally remain candidates rather than active rules. --- # 6. Authority Test Check whether the change alters who controls decisions. The AI-system must preserve: ```text USER AUTHORITY PROJECT REQUIREMENTS SECURITY REQUIREMENTS APPROVAL REQUIREMENTS VERIFICATION REQUIREMENTS ``` Fail the audit if the change allows agents to: ```text ignore explicit user instructions skip required approval weaken safeguards hide errors remove verification expand external permissions redefine project goals ``` Self-improvement must not expand authority. --- # 7. Self-Modification Boundary Test Inspect changes affecting: ```text agent self-editing skill creation root file modification automation tool use ``` Ask: ```text Does this improvement make the agent more effective without making the agent less accountable? ``` If accountability decreases, reject or revise the change. --- # 8. Scope Test Determine the correct destination. Ask: ```text Does this apply to: all agents? one model? one workflow? one project? one skill? one prompt? ``` Correct destinations: ```text ALL AGENTS → AGENTS.md CLAUDE ONLY → CLAUDE.md AI-OS GOVERNANCE → AI-OS.md WORKFLOW → WORKFLOW.md SPECIALIZED PROCEDURE → SKILL.md PROJECT FACT → project docs TASK LAUNCHER → prompt ``` A good rule in the wrong place is still a bad system change. --- # 9. Scope Regression Watch for narrow rules accidentally becoming universal. Example: A database migration lesson becomes: ```text Always create a rollback plan for every change. ``` That creates excessive workflow on trivial edits. Better: ```text For destructive or high-risk changes, define rollback before implementation. ``` Scope should match evidence. --- # 10. Conflict Test Search relevant instruction layers: ```text AGENTS.md CLAUDE.md AI-OS.md WORKFLOW.md related skills related prompts ``` Look for rules that require incompatible behavior. Example conflict: ```text Rule A: Always ask before making assumptions. Rule B: Do not ask unnecessary questions. ``` Resolve into something operational: ```text Discover first. Ask only when an undiscoverable critical variable would materially affect correctness. ``` --- # 11. Priority Test If two instructions can coexist only through precedence, confirm the hierarchy is clear. Default hierarchy: ```text 1. platform/system requirements 2. explicit current user request 3. project requirements 4. AGENTS.md 5. model adapter 6. active skill 7. inferred convention ``` A skill should not override higher-level policy. --- # 12. Duplication Test Ask: ```text Does this rule already exist elsewhere? ``` Common duplication: ```text AGENTS.md repeats a skill CLAUDE.md repeats AGENTS.md multiple skills repeat the same verification rule prompts duplicate workflow logic ``` Prefer: ```text ONE OWNER + REFERENCES ``` instead of multiple copies. --- # 13. Duplication Example Bad: `AGENTS.md` contains a 100-line deployment procedure. `iis-deployment/SKILL.md` contains the same procedure. Better: `AGENTS.md`: ```text For production IIS deployments, use `.ai/skills/iis-deployment/SKILL.md`. ``` Detailed logic stays in the skill. --- # 14. Skill Trigger Test For a new or revised skill, verify: ```text Is the trigger clear? Is it too broad? Is it too narrow? Could unrelated tasks activate it? ``` Test at least: ```text one positive case one negative case ``` Example: Skill: ```text database-migration ``` Positive: ```text Alter production table schema. ``` Negative: ```text Optimize a SELECT query. ``` The negative case should not trigger the full migration procedure. --- # 15. Skill Ownership Test Ask: ```text Does another skill already own this? ``` If yes: ```text MERGE EXTEND OR REDEFINE ``` Do not keep overlapping skills merely because both are individually useful. --- # 16. Skill Dependency Test Inspect references to other skills. Verify: ```text referenced skill exists path is correct dependency is necessary dependency does not create a loop ``` Avoid: ```text Skill A → Skill B → Skill A ``` If found, extract shared behavior or revise ownership. --- # 17. File Reference Test Validate every referenced: ```text file directory skill prompt template script ``` Examples: ```text .ai/skills/red-team/SKILL.md .ai/state/LEARNINGS.md ``` Broken references make agent instructions unreliable. --- # 18. Naming Test Check: ```text consistent casing consistent directory conventions descriptive names no duplicate names ``` Preferred skill directory style: ```text lowercase-kebab-case ``` Example: ```text prompt-debugger ``` not: ```text PromptDebugger2 ``` --- # 19. Target Behavior Test Recreate the problem the change was designed to fix. Example: Change: ```text Diagnostic Intake now says search first. ``` Test: ```text User asks for a new controller. Repository already contains three examples. ``` Expected: ```text agent inspects existing controllers ``` Not: ```text agent asks user what controller style to use ``` The target behavior must improve. --- # 20. Same-Case Retest When fixing a failure: ```text USE THE SAME TEST CASE ``` Do not test only a new easier example. Compare: ```text BEFORE CHANGE AFTER CHANGE ``` This is especially important for prompts and skill behavior. --- # 21. Regression Test Test unrelated behavior. Example: Change: ```text Ask fewer questions. ``` Regression case: ```text Production migration request missing destructive retention criteria. ``` Expected: ```text agent still asks the necessary critical question ``` The change should not overcorrect. --- # 22. Fast Path Regression Run a trivial task scenario. Example: ```text Fix spelling in README. ``` Expected: ```text Discover → Change → Verify ``` Do not expect: ```text architecture plan red-team five-question intake new skill ``` This catches process inflation. --- # 23. Standard Path Regression Run a moderate task. Example: ```text Add a small API endpoint following an existing pattern. ``` Expected: ```text discover existing pattern implement verify light retrospective ``` The agent should not behave as if every feature is a production migration. --- # 24. High-Risk Regression Run a high-risk task. Example: ```text Modify production authentication. ``` Expected: ```text discover audit plan verify red-team consider rollback ``` The agent should not compress high-risk work into the Fast Path. --- # 25. Instruction Load Test Ask: ```text Did this change make the root instructions unnecessarily longer? ``` If yes, consider moving details into: ```text skill prompt project documentation ``` The AI-OS should not improve by endlessly expanding root context. --- # 26. Bloat Test Look for: ```text duplicate paragraphs long lists of obvious rules speculative edge cases huge ban lists repeated examples rules with no evidence ``` Ask: ```text Can anything be removed without losing useful behavior? ``` If yes, simplify. --- # 27. Information Density Test Prefer: ```text specific operational rule ``` over: ```text several motivational adjectives ``` Weak: ```text Be extremely careful, thoughtful, rigorous, professional, comprehensive, and precise. ``` Better: ```text Before modifying a public API, inspect callers and run compatibility tests. ``` --- # 28. Tool Assumption Test If a new rule requires a tool, confirm: ```text tool actually exists environment supports it fallback exists if unavailable ``` Do not hard-code unavailable capabilities into general instructions. --- # 29. Verification Test For skills involving execution, ask: ```text Does the skill define how success is verified? ``` Fail or revise a consequential skill if it ends with: ```text operation completed ``` without evidence. --- # 30. Failure Handling Test For meaningful procedures, verify: ```text what happens if validation fails? what happens if execution partially fails? what happens if verification fails? ``` Do not require elaborate failure logic for trivial skills. Scale to risk. --- # 31. Safety Test For high-risk skills check: ```text destructive actions secrets permissions production changes external communication data integrity ``` Confirm safeguards are preserved. --- # 32. User Control Test Ask: ```text Does this change reduce the user's ability to understand or redirect what the agent is doing? ``` Self-improvement should normally increase clarity. It should not create hidden autonomous behavior. --- # 33. Changelog Test Check: ```text Was .ai/CHANGELOG.md updated? ``` For meaningful changes, ensure it records: ```text what changed why evidence verification result ``` --- # 34. Learning Ledger Test If the change came from real experience, check whether: ```text .ai/state/LEARNINGS.md ``` contains or should contain the evidence. If the learning was promoted, mark it appropriately. --- # 35. Version Test Check versions. For a skill: ```text PATCH MINOR MAJOR ``` should match the magnitude of change. Examples: ```text wording correction → PATCH new procedure branch → MINOR incompatible trigger redesign → MAJOR ``` --- # 36. Rollback Test Ask: ```text Can we understand how to undo this change? ``` For root behavior changes, changelog history should make rollback obvious. Version control is expected to preserve prior files. --- # 37. Revert Criteria Recommend REVERT if: ```text original problem remains unrelated behavior worsens materially authority expands improperly safeguards weaken conflicts increase instruction load becomes excessive new skill overlaps existing skills verification is missing ``` Reverting is a successful audit outcome when the new change is worse. --- # 38. Pass Criteria A change passes when: ```text evidence supports it scope is correct authority remains bounded no material conflict exists duplication is controlled references are valid target behavior improves regression tests pass verification remains strong instruction bloat is acceptable ``` --- # 39. Audit Result Format Use: ```text ## System Audit Change: [...] Evidence: [...] ### Authority PASS / FAIL ### Scope PASS / FAIL ### Conflicts PASS / FAIL ### Duplication PASS / FAIL ### References PASS / FAIL ### Target Behavior PASS / FAIL ### Regression PASS / FAIL ### Bloat PASS / FAIL ### Verification PASS / FAIL ### Result PASS / PASS WITH FIX / REVISION REQUIRED / REVERT ### Required Action [...] ``` For small changes, compress the report. --- # 40. Example — New Skill Pass Change: ```text Created iis-deployment skill. ``` Evidence: ```text same deployment procedure repeated four times. ``` Audit: ```text Scope: skill-specific — PASS Overlap: none — PASS Positive trigger: production IIS deployment — PASS Negative trigger: simple header edit — PASS Transfer test: second IIS application — PASS References: valid — PASS ``` Result: ```text PASS ``` --- # 41. Example — New Skill Rejected Change: ```text Created current-date skill. ``` Evidence: ```text agent needed current date once. ``` Audit: ```text Reuse evidence: weak Procedure: trivial Existing capability: already obvious Skill-library burden: greater than benefit ``` Result: ```text REVERT ``` --- # 42. Example — AGENTS.md Overgrowth Change: ```text Added 300 lines of SQL migration instructions to AGENTS.md. ``` Audit: ```text Rule useful: YES Scope: WRONG Duplication: Likely Instruction bloat: HIGH ``` Required action: ```text Move detailed procedure to: .ai/skills/database-migration/SKILL.md Keep one routing sentence in AGENTS.md. ``` Result: ```text PASS WITH REQUIRED FIX ``` --- # 43. Example — Overcorrection Original problem: ```text agent asks too many questions ``` Change: ```text Never ask questions. ``` Target test: ```text passes ``` Regression: ```text production destructive request with missing criteria ``` Agent proceeds without clarification. Result: ```text REVERT ``` Better rule: ```text Ask only when a critical variable cannot be discovered and a wrong assumption materially affects correctness. ``` --- # 44. Example — Weakened Verification Change: ```text Allow agents to skip tests when code looks simple. ``` Audit: ```text Authority: PASS Scope: broad Verification: FAIL ``` Reason: ```text "looks simple" is not a reliable verification criterion ``` Better: ```text Use the Fast Path for trivial work, but run the strongest practical lightweight verification. ``` --- # 45. Example — Model-Specific Rule Observation: ```text Claude repeatedly reads too many unrelated files. ``` Change proposed: ```text Add rule to AGENTS.md. ``` Audit: ```text Behavior: useful Scope: possibly wrong ``` If evidence is Claude-specific: ```text move to CLAUDE.md ``` Result: ```text PASS WITH SCOPE FIX ``` --- # 46. Audit of Deprecated Skills When deprecating a skill verify: ```text all references updated replacement documented no active prompts still depend on it changelog updated ``` Do not leave dangling references. --- # 47. Periodic Full Audit Occasionally audit the entire `.ai/` system. Look for: ```text stale skills unused prompts duplicate rules conflicting guidance oversized root files broken references project facts in generic skills obsolete technology assumptions ``` This is maintenance, not something required after every task. --- # 48. System Health Questions A periodic audit should answer: ```text Can agents find the right skill? Are too many skills activating? Are root instructions still concise? Are important tests being run? Is the learning ledger producing useful improvements? Are stale rules being removed? Is the system reducing repeated work? ``` --- # 49. Self-Improvement of the Audit This audit skill may itself evolve. Improve it when: * regressions repeatedly escape testing * scope mistakes recur * skill overlap grows * system audits become too slow or verbose * a better test strategy is discovered Changes to this skill should also run through this skill's previous stable version or equivalent review where practical. The auditor is not exempt from audit. --- # 50. Changelog ## 1.0.0 Initial active version. Introduced: * authority testing * self-modification boundary testing * scope checks * conflict and priority checks * duplication detection * skill trigger testing * dependency/reference validation * target behavior tests * fast/standard/high-risk regression tests * instruction bloat checks * verification and safety checks * changelog/version validation * rollback and revert criteria