Meta-Prompting AI Prompts

Step 1: Define the task and failure modes — describe the extraction or analysis task precisely. List the 5 most common ways the model currently fails on this task (wrong format, wrong field, missed edge case, wrong inference, etc.). Step 2: Identify example coverage needs — for each failure mode, determine what kind of example would teach the model to handle it correctly. The example set should cover: a clean/easy case, a hard/ambiguous case, an edge case for each common failure mode, and a 'correct refusal' case where the answer is null or unknown. Step 3: Draft examples — write input-output pairs for each required example type. For each example: choose the simplest input that demonstrates the pattern (complex examples obscure the lesson), write the exact correct output in the target format, and add a brief comment explaining what this example teaches (this comment is for you, not the model). Step 4: Order the examples — order them from simplest to most complex. Studies show that example order affects LLM performance. The first example anchors the model's interpretation of the task; make it the clearest, most typical case. Step 5: Test individual examples — before assembling into a full prompt, test each example by asking the model to predict the output without seeing the answer. If the model gets it right without the example, the example may not be needed. If the model gets it wrong, the example is teaching something valuable. Step 6: Assemble and evaluate — combine the examples into the prompt and run the full evaluation suite. Compare performance with 0, 2, 4, 6, and 8 examples to find the optimal number. More is not always better — irrelevant examples add noise. Step 7: Document the example set — for each example, record: why it was included, what failure mode it addresses, and when it should be updated. Treat examples as code: version-controlled, with change history and rationale.

Design a meta-prompt that uses an LLM to automatically improve a data extraction or analysis prompt based on observed failures. Manual prompt tuning is iterative and intuition-driven. Automated prompt optimization uses the model's own reasoning to generate improvements systematically. 1. The optimization loop: Step 1 — Failure collection: Run the current prompt on the evaluation dataset. Collect all cases where the output failed (wrong extraction, schema violation, incorrect analysis). Step 2 — Failure analysis meta-prompt: 'You are a prompt engineer. Here is a prompt that is failing on certain inputs: [CURRENT PROMPT] Here are the inputs where it failed and what the correct output should have been: [FAILURE CASES WITH EXPECTED OUTPUTS] Analyze the failure pattern: 1. What is the common characteristic of all failing inputs? 2. What aspect of the prompt is causing these failures? (unclear instruction, missing edge case handling, wrong example, etc.) 3. Propose a specific, minimal change to the prompt that would fix these failures without breaking passing cases.' Step 3 — Candidate prompt generation: Generate 3–5 candidate improvements based on the failure analysis. Step 4 — Candidate evaluation: Run each candidate prompt on the full evaluation dataset. Select the prompt with the highest overall pass rate that does not regress previously passing cases. Step 5 — Iterate: Repeat steps 1–4 until pass rate plateaus or meets the target. 2. Guardrails for automated optimization: - Require human review before deploying any auto-optimized prompt to production - Never optimize on the same dataset used for evaluation (overfitting risk) - Track prompt version history: keep all previous versions and their eval scores - Limit prompt length growth: if the optimized prompt is > 50% longer than the original, require human review 3. What automated optimization cannot do: - It cannot fix failures caused by genuinely ambiguous instructions without human clarification - It cannot improve performance beyond the model's capability ceiling - It is not a substitute for a well-curated evaluation dataset Return: the failure analysis meta-prompt, optimization loop implementation, candidate evaluation framework, and a worked example showing 3 iterations of improvement on a real extraction prompt.