Here is the abstract and summary for the provided blog post and its discussion.
Abstract:
This article details an experiment that significantly improved the performance of a smaller language model, GPT-5-mini, on the Tau² benchmark for agentic tasks. Facing a low baseline success rate of 55% in the "telecom" domain, the author used a more advanced model (Claude) to automatically rewrite and restructure the agent's guiding policies. The rewrite transformed verbose, descriptive text into clear, actionable, and structured instructions, incorporating elements like decision trees, explicit tool-call syntax, and step-by-step verification logic. This prompt optimization resulted in a 22.7% increase in the success rate to 67.5% and halved the number of unsolvable tasks. The key takeaway is that strategic prompt engineering—specifically making prompts more programmatic—can unlock substantial performance gains in smaller, more efficient models, making them viable alternatives to their larger, more costly counterparts.
Boosting GPT-5-mini Performance Through Prompt Engineering
Baseline Performance: The experiment begins by establishing a baseline for the GPT-5-mini model on the telecom_small subset of the Tau² benchmark. The initial result was a low success rate of 55%, significantly underperforming the flagship GPT-5 model's 97% accuracy.
The "Hack" - Automated Prompt Rewriting: To improve performance, the author used the Claude language model to analyze and rewrite the AI agent's core policies. The goal was to make the instructions clearer and easier for a less powerful model to follow.
Key Prompt Improvements: The rewritten policies moved from narrative descriptions to structured, programmatic instructions. Key changes included:
Structure & Flow: Using decision trees and numbered, sequential steps.
Clarity: Providing exact tool function names and parameters, using binary (yes/no) conditions, and specifying error handling.
Cognitive Load Reduction: Adding quick-reference tables and highlighting common patterns and mistakes to avoid.
Actionable Language: Replacing verbose text with imperative commands like "Check X" and "If Y then Z."
The Result - A 22% Accuracy Boost: The optimized prompts led to a significant performance increase.
The success rate (pass^k=1) jumped from 55% to 67.5%, a 22.7% improvement.
The model's reliability on retries also improved.
The number of tasks that consistently failed dropped by 50% (from 6 to 3).
Main Takeaway: The experiment demonstrates that smaller, more efficient LLMs can achieve high performance when guided by well-structured, unambiguous, and programmatic prompts. Using a frontier model to automate the optimization of these prompts is an effective strategy to make smaller models a compelling and cost-effective alternative.
Community Feedback: In response to reader requests, the author updated the post with a link to a Pull Request showing the exact "before and after" changes to the agent policies.
Summary of Hacker News Discussion
The Hacker News discussion provided additional context, critiques, and related insights on the article's findings.
Context on the Benchmark: An OpenAI employee explained that the "Telecom" benchmark is emphasized because it is more robust than older domains like "retail" and "airline." It grades tasks based on final outcomes rather than matching a single, brittle reference solution, providing a cleaner signal of an agent's capability.
"Reinventing Programming": A dominant theme was that the "optimized" prompt techniques—using decision trees, clear steps, and imperative commands—are essentially a reinvention of programming principles in natural language. Commenters noted the shift from vague instructions to writing "code" for the LLM.
Critiques and Skepticism: Some users questioned whether the results were truly generalizable or if the prompt was simply overfitted to the benchmark. A key concern was that making instructions too explicit might remove the "agentic" reasoning the benchmark is supposed to measure, effectively teaching the model the answers to the test.
Connection to Existing Frameworks: Several commenters pointed out that these concepts of programmatic prompt optimization are not new and have been formalized in frameworks like DSPy, which is designed to compile natural language instructions into optimized prompts for LLMs.
Requests for Transparency: Initially, many users criticized the article for not including the actual "before and after" prompts. This feedback led directly to the author updating the post with a link to the code changes, which was well-received.