18KB
Prompt Debugger
Version: 1.0.0 Status: ACTIVE
Purpose
Diagnose why a prompt, system instruction, agent workflow, or reusable AI process produced poor output.
The goal is not:
FAILED OUTPUT
↓
ADD MORE WORDS TO PROMPT
The goal is:
FAILED OUTPUT
↓
IDENTIFY FAILURE CLASS
↓
FIND ROOT CAUSE
↓
MAKE SMALLEST REPAIR
↓
RETEST
Prompt debugging should behave more like software debugging than creative rewriting.
1. Trigger Conditions
Use this skill when:
- a prompt repeatedly produces poor output
- an agent ignores important instructions
- output formatting drifts
- the AI asks unnecessary questions
- the AI makes unsupported assumptions
- the AI is too verbose
- the AI is too brief
- the AI chooses the wrong workflow
- the AI ignores tools it should use
- the AI uses tools unnecessarily
- the AI produces technically correct but unusable results
- prompt quality varies significantly between similar inputs
- a system instruction causes unintended behavior elsewhere
- a prompt change needs regression testing
Do not use this skill when:
- the underlying problem is actually missing project documentation
- the failure was caused by bad input data
- the model lacked a required tool or permission
- the user request itself changed
- the issue is a one-time harmless variation
First determine whether the prompt is actually the problem.
2. Core Debugging Principle
Preserve evidence.
When a prompt fails, retain:
ORIGINAL PROMPT
INPUT
OUTPUT
EXPECTED OUTPUT
ENVIRONMENT / MODEL
TOOLS AVAILABLE
RELEVANT SYSTEM INSTRUCTIONS
Do not immediately overwrite the failed version.
Without the original evidence, debugging turns into guessing.
3. Prompt Debugging Loop
Use:
REPRODUCE
↓
CLASSIFY
↓
ISOLATE
↓
HYPOTHESIZE
↓
PATCH
↓
RETEST SAME CASE
↓
REGRESSION TEST
↓
PROMOTE OR REVERT
4. Reproduce the Failure
Before changing the prompt, determine whether the failure is repeatable.
Ask:
Does the same input produce the same problem?
Does a similar input produce the same problem?
Is the problem model-specific?
Is the problem caused by missing context?
If the failure cannot be reproduced, classify confidence appropriately.
Do not create permanent rules from weak evidence.
5. Define Expected Behavior
You cannot debug a prompt without knowing what success should look like.
Define:
EXPECTED BEHAVIOR
ACTUAL BEHAVIOR
DIFFERENCE
Example:
Expected:
Ask only one diagnostic question.
Actual:
Asked eight questions at once.
Difference:
Interaction mode and question-count constraint were not enforced.
6. Failure Classes
Check these failure classes before modifying the prompt.
1. MISSING CONTEXT
2. AMBIGUOUS OBJECTIVE
3. WEAK CONSTRAINTS
4. CONFLICTING INSTRUCTIONS
5. INSTRUCTION OVERLOAD
6. WRONG INTERACTION MODE
7. POOR OUTPUT CONTRACT
8. INCORRECT TOOL ASSUMPTIONS
9. STALE PROJECT INFORMATION
10. WRONG SCOPE
11. WEAK VERIFICATION
12. BAD EXAMPLE / FEW-SHOT BIAS
13. PERSONA FAILURE
14. EXCESSIVE AUTONOMY
15. EXCESSIVE CLARIFICATION
16. PROMPT DOES TOO MANY JOBS
7. Failure Class — Missing Context
Symptoms:
generic output
invented assumptions
wrong architecture
wrong technology
questions that repository evidence could answer
Example:
Prompt:
Create a controller for this project.
but no project context is supplied or discovered.
The model invents a framework pattern.
Root cause:
MISSING CONTEXT
Possible repair:
Before implementing, inspect existing controllers,
routing, project instructions, and related services.
Follow the established project pattern.
Do not solve missing context by adding generic architecture advice.
8. Failure Class — Ambiguous Objective
Symptoms:
technically valid output
wrong level of detail
wrong deliverable
agent solves a different problem
Example:
Improve this script.
“Improve” could mean:
faster
safer
shorter
more readable
more compatible
Repair the objective.
Example:
Improve reliability and error reporting without changing
the script's public behavior or adding dependencies.
9. Failure Class — Weak Constraints
Symptoms:
unwanted framework
unnecessary dependencies
unrelated refactor
changed public behavior
overengineering
Example failure:
User wanted one ASP Classic endpoint.
Agent introduces a new application framework.
Repair:
Follow the existing ASP Classic architecture.
Do not introduce a new framework.
Do not modify unrelated endpoints.
Use constraints that protect real boundaries.
10. Failure Class — Conflicting Instructions
Symptoms:
inconsistent output
agent follows one rule while violating another
behavior changes unpredictably
Example:
Instruction A:
Always ask before making assumptions.
Instruction B:
Do not ask unnecessary questions.
These may conflict.
Repair by defining hierarchy:
Discover first.
If a critical variable cannot be discovered and
a wrong assumption materially affects correctness, ask.
Otherwise state a safe assumption and proceed.
Resolve the conflict rather than adding a third vague rule.
11. Failure Class — Instruction Overload
Symptoms:
important rules ignored
output becomes generic
model follows later minor rules instead of important earlier rules
behavior becomes inconsistent
Possible cause:
too many permanent instructions
Before adding more instructions, consider:
REMOVE
MERGE
MOVE TO SKILL
MOVE TO PROJECT DOCS
Prompt debugging may require deleting instructions.
12. Failure Class — Wrong Interaction Mode
Symptoms:
agent asks questions when complete output was requested
agent gives full answer when teaching was requested
agent waits unnecessarily
agent produces partial output for an autonomous task
Classify the required mode:
AUTONOMOUS
INTERACTIVE
SOCRATIC
REVIEW-ONLY
ADVERSARIAL
Then specify it explicitly when needed.
13. Failure Class — Poor Output Contract
Symptoms:
unstructured response
missing fields
hard-to-use output
inconsistent format
extra commentary
Weak:
Review this.
Better:
Return:
1. blockers
2. important findings
3. optional improvements
For each finding:
- location
- issue
- consequence
- recommended fix
Output contracts should make downstream use easier.
14. Failure Class — Incorrect Tool Assumptions
Symptoms:
agent claims something cannot be checked
agent invents results
agent searches the web when project files contain the answer
agent fails to run available tests
Repair tool behavior.
Example:
Search the repository before stating that a function does not exist.
Run the existing test command after modifying code.
Do not list every possible tool.
Define what evidence must be gathered.
15. Failure Class — Stale Project Information
Symptoms:
wrong framework version
wrong deployment target
obsolete directory structure
old command
retired API
The prompt may be correct but project context is stale.
Repair the source of truth.
Possible destination:
PROJECT DOCUMENTATION
not necessarily the prompt.
16. Failure Class — Wrong Scope
Symptoms:
A rule intended for one domain affects every task.
Example:
A prompt rule says:
Always create a rollback plan.
This may be useful for:
database migrations
deployments
infrastructure
but excessive for:
spelling correction
Move narrow procedures into skills.
17. Failure Class — Weak Verification
Symptoms:
good-looking output
unverified commands
wrong code
silent data problems
Repair:
After implementation, run the strongest available verification.
If verification cannot be performed,
state exactly what remains unverified.
Verification requirements often improve prompt reliability more than additional stylistic rules.
18. Failure Class — Bad Example Bias
Examples strongly influence behavior.
A few-shot example may accidentally teach:
wrong structure
wrong assumptions
wrong verbosity
project-specific details
If prompt output keeps resembling an example too literally, inspect the examples.
Ask:
Is the model learning the intended rule
or copying accidental details?
Remove or diversify biased examples.
19. Failure Class — Persona Failure
Symptoms:
reviewer too agreeable
reviewer too hostile
fake objections
wrong priorities
Weak persona:
Be a tough reviewer.
Better:
You are the infrastructure engineer responsible
for recovering this deployment during an outage.
Primary fear:
irreversible deployment failure.
Non-negotiable metric:
recoverability.
Persona should encode incentives.
20. Failure Class — Excessive Autonomy
Symptoms:
agent makes major architectural decisions without surfacing them
agent performs destructive changes
agent modifies unrelated systems
agent expands scope
Repair authority boundaries.
Example:
Routine implementation decisions may proceed autonomously.
Surface decisions that alter architecture,
security boundaries, destructive behavior,
or public compatibility.
21. Failure Class — Excessive Clarification
Symptoms:
agent repeatedly asks questions
user has already supplied information
questions could be answered from repository evidence
task becomes stalled
Repair using Diagnostic Intake rules.
Example:
Search project evidence before asking.
Ask only when a critical variable cannot be discovered
and a wrong assumption materially affects correctness.
22. Failure Class — Prompt Does Too Many Jobs
Symptoms:
very long prompt
many unrelated responsibilities
poor adherence
complex branching
hard to test
Example:
One prompt attempts to:
design architecture
write code
review security
teach the user
write documentation
create release notes
audit itself
Consider splitting into:
SKILLS
WORKFLOW PHASES
TASK PROMPTS
Composition is often more reliable than one giant prompt.
23. Root Cause Table
For meaningful prompt debugging, use:
SYMPTOM | FAILURE CLASS | EVIDENCE | REPAIR
Example:
Agent asks 12 questions
Excessive clarification
Repository already contained most answers
Add discovery-first intake rule
Agent overengineers simple fixes
Wrong scope
Architecture rules trigger on all tasks
Introduce Fast/Standard/High-Risk paths
Agent ignores critical instructions
Instruction overload
Root prompt exceeds useful scope
Move procedures to skills
24. Make the Smallest Repair
After identifying the root cause:
DO NOT REWRITE EVERYTHING
Change only what is necessary.
Example:
Bad approach:
replace 500-line system prompt
Preferred:
add one missing output rule
or:
remove one conflicting instruction
or:
move one procedure into a skill
Small changes are easier to test.
25. Retest the Same Input
Always retest the exact failure case first.
Use:
SAME PROMPT CONTEXT
SAME INPUT
CHANGED INSTRUCTION
Compare:
BEFORE
AFTER
If the original problem remains, the hypothesis was probably wrong or incomplete.
26. Regression Test
After the failed case improves, test unrelated behavior.
Example:
Prompt change:
Ask fewer questions.
Regression test:
Give it a genuinely ambiguous high-risk migration.
Ensure it still asks when clarification really is required.
Every prompt rule has tradeoffs.
27. A/B Comparison
When practical, compare:
VERSION A — CURRENT
VERSION B — REVISED
using several representative inputs.
Evaluate:
correctness
instruction adherence
usefulness
verbosity
number of manual corrections
verification quality
Avoid choosing the revised version merely because it is newer.
28. Prompt Test Set
Important production prompts may maintain a small test suite.
Example:
tests/
├── normal.md
├── ambiguous.md
├── edge-case.md
├── incomplete-context.md
└── regression.md
Expected behavior can be documented in each case.
This makes prompt changes more like code changes.
29. Prompt Unit Test Concept
A prompt test may assert behavior such as:
GIVEN:
A one-line typo request
EXPECT:
Fast Path
DO NOT EXPECT:
architecture plan
red-team review
five clarification questions
Another:
GIVEN:
Production database migration with missing backup requirements
EXPECT:
High-Risk Path
rollback discussion
critical clarification if not discoverable
Behavioral tests are often enough.
30. Detect Instruction Bloat
When debugging a large prompt, ask:
Can this rule be removed?
Can two rules be merged?
Does this belong in a skill?
Does this belong in project documentation?
Does this apply often enough to remain global?
Deleting low-value instructions can improve adherence to important ones.
31. Conflict Search
When root instructions change, inspect other instruction layers:
AGENTS.md
CLAUDE.md
AI-OS.md
WORKFLOW.md
loaded skills
project docs
Look for contradictory behavior.
Example:
AGENTS.md:
Compress workflow for trivial tasks.
Skill:
Always perform full architecture audit.
That conflict should be corrected.
32. Prompt Failure Postmortem
For recurring or important failures, record:
WHAT HAPPENED
EXPECTED BEHAVIOR
ROOT CAUSE
WHY EXISTING RULES FAILED
CHANGE MADE
TEST PERFORMED
RESULT
If reusable, capture the learning in:
.ai/state/LEARNINGS.md
33. Example — Too Much Fluff
Problem:
Generated status reports contain:
lengthy introduction
corporate buzzwords
repeated conclusion
Bad repair:
Add 50 banned words.
Better diagnosis:
Failure class:
weak output/tone constraint.
Repair:
Start with the substantive update.
Use direct language.
Prefer concrete facts.
Remove redundant closing summaries.
Then optionally use:
.ai/skills/coffee-chat-editor/SKILL.md
34. Example — Wrong Architecture
Problem:
Agent repeatedly creates new abstractions for minor features.
Diagnosis:
WRONG SCOPE
+
MISSING PROJECT CONTEXT
Repair:
Search for analogous existing implementations first.
Prefer existing architecture.
Do not introduce abstractions unless the task demonstrates repeated need.
Test on:
small feature
large new subsystem
Ensure the second can still justify abstraction.
35. Example — Too Many Questions
Problem:
User requests a script.
Agent asks:
OS version?
PowerShell version?
domain?
permissions?
directory?
environment?
even though most details are discoverable or irrelevant.
Diagnosis:
EXCESSIVE CLARIFICATION
Repair:
Ask only for critical variables that cannot be discovered.
Use safe assumptions for low-risk reversible details.
36. Example — Agent Skips Testing
Problem:
Agent says:
The build should pass.
without running it.
Diagnosis:
WEAK VERIFICATION
Repair:
When a build command is available, execute it after relevant changes.
Do not claim successful verification based on code inspection alone.
37. Example — Prompt Too Large
Problem:
Root prompt contains:
coding rules
deployment instructions
writing style
database migration steps
teaching behavior
security checklist
API design
release procedures
Agents inconsistently follow important rules.
Diagnosis:
INSTRUCTION OVERLOAD
Repair:
Keep model-neutral core behavior in AGENTS.md.
Move detailed procedures into skills.
Load skills when their triggers match.
This is structural prompt debugging.
38. Example — Overly Critical Red Team
Problem:
Red-team output contains dozens of speculative objections.
Diagnosis:
PERSONA FAILURE
+
POOR OUTPUT CONTRACT
Repair:
Require:
1–3 highest-impact findings
severity
evidence
real consequence
mitigation
distinguish speculative concerns
39. Example — Teaching Agent Gives Answers Too Soon
Problem:
Agent explains a concept, asks a question, and immediately gives the answer.
Diagnosis:
WRONG INTERACTION MODE
Repair:
After asking the drill, stop.
Do not provide the answer until the learner responds
or explicitly asks for it.
40. Self-Improvement Decision
After fixing a prompt, ask:
Was this failure one-off?
Does it affect one prompt?
Does it affect a skill?
Does it affect all agents?
Does it apply only to one model?
Then place the repair in the narrowest correct location.
41. Promotion Rules
Promote a prompt fix when:
failure is reproducible
root cause is reasonably understood
new behavior performs better
regression tests pass
Do not promote merely because one response looked better.
42. Reversion
Revert the prompt change when:
original failure remains
unrelated behavior becomes worse
instruction conflict increases
prompt becomes materially harder to follow
manual correction increases
Record meaningful failed experiments in the changelog.
43. Output Contract for This Skill
A prompt-debugging result should normally contain:
OBSERVED FAILURE
EXPECTED BEHAVIOR
FAILURE CLASS
ROOT CAUSE
SMALLEST REPAIR
RETEST
REGRESSION TEST
PROMOTION DECISION
For trivial issues, compress this.
44. Self-Improvement
Improve this skill when:
- new recurring failure classes emerge
- certain failure classes overlap unnecessarily
- debugging becomes too heavyweight
- regression testing misses predictable problems
- a better prompt test strategy emerges
Avoid turning every individual prompt failure into a new general failure class.
45. Changelog
1.0.0
Initial active version.
Introduced:
- evidence-preserving prompt debugging
- failure classification
- root-cause diagnosis
- minimal prompt repair
- same-case retesting
- regression testing
- instruction-bloat detection
- conflict analysis
- prompt test-suite concept
- promotion and reversion rules