Prompt engineering didn't disappear — it got absorbed into a bigger discipline. Here's what actually matters now, and the eval habit that replaces "finding the right wording."
In 2023, "prompt engineer" was floated, semi-seriously, as a six-figure job that required no computer science degree — just a knack for finding the magic words that made a model behave. Job boards filled up with the title. Twitter threads promised "10 prompts that will change your life." An entire cottage industry of prompt marketplaces appeared, selling text files for money.
That title has now mostly vanished from job postings. Not because the underlying work stopped mattering — but because the work changed shape so much that the old name stopped fitting.
Early models were genuinely brittle. Rephrasing a request — adding "think step by step," switching from a question to an instruction, capitalizing certain words — could swing output quality dramatically. In that environment, knowing the right phrasing was a real skill, and a scarce one.
Newer models are far more robust to phrasing. The same underlying intent, expressed in a dozen different ways, now produces results that are far more consistent. Some of the "best practices" from 2023 — elaborate persona-setting, specific magic phrases, formatting tricks — measurably stopped making a difference somewhere in the last eighteen months. The skill of finding the perfect wording got commoditized at roughly the same rate models got better at understanding plain ones.
If you've been following the last three issues, you already know where this is going. Issue 001 was about context windows becoming enormous. Issue 002 was about retrieval architecture. Issue 003 was about agent failure modes. Each of those is, in its own way, about the same underlying shift: the hard problem moved from "what words do I use?" to "what does the model have access to, structured how, and how do I know if it's working?"
Call it context engineering, call it AI systems design — the name matters less than the shape of the work. It now looks like:
Context design. Deciding what information the model sees, in what order, with what structure — directly informed by how attention actually behaves across long contexts (see Issue 002's "lost in the middle" discussion).
Output structure. Function calling and structured outputs are now table stakes across every major provider. The skill isn't writing a clever prompt that produces JSON — it's designing the schema itself, and validating against it (Issue 003, failure mode 2).
Evaluation. This is the big one, and it's the one most individuals skip. The honest answer to "is this prompt good?" was, for most of 2023 and 2024, vibes — try it a few times, see if it looks right, ship it. That doesn't scale, and it actively misleads you: a prompt that looks great on the three examples you tried can fail silently on the fourth.
If your job title was "prompt engineer," your job didn't disappear — it got renamed to "AI systems engineer" or "applied AI engineer," and it got considerably more interesting. If you're learning AI skills from scratch, the highest-leverage thing to learn isn't a library of clever phrasings. It's how to build a small evaluation set for whatever you're building, so that when you change a prompt, a model, or a piece of context, you have a number — not a feeling — that tells you whether it got better or worse.
That habit is this issue's Tool Drop and Prompt, below.
Promptfoo is an open-source command-line tool for testing prompts the same way you'd test code: define a set of test cases, define what a passing result looks like, run it against one or more models, and get a pass/fail grid back. It's the most direct, no-frills implementation of "evals" I've used.
The setup is a YAML file: a prompt template, a list of test inputs, and a list of assertions per test (contains a string, matches a regex, passes an LLM-graded rubric, or — if you want to get serious — a custom function). Run it, and you get a side-by-side comparison across however many prompt variants or models you want to test at once.
Why this matters more than it sounds: the moment you have even five test cases in a file, "should I change this prompt?" stops being a debate and becomes a command you run. I've caught regressions from a one-word prompt change that I never would have noticed by eye — the kind of silent quality drift that's invisible until a user complains.
This paper makes a simple but uncomfortable point: when researchers report "Model X scores 87% on Benchmark Y," that number almost always comes from one specific prompt template. The authors re-ran standard benchmarks across dozens of semantically equivalent prompt variations — same question, different phrasing — and found the resulting scores swung by double-digit percentage points depending on which prompt happened to be chosen.
In other words: a meaningful chunk of the difference between "this model is great" and "this model is mediocre" in published benchmarks isn't the model at all. It's which of many roughly-equivalent prompts the researchers happened to type first.
The paper's recommendation is exactly the shift this issue describes: stop treating any single prompt as the source of truth, and evaluate across a distribution of prompts that express the same intent. A system that's robust across phrasing is a better system than one that's brittle but happens to score well on the phrasing someone tested.
Fittingly, this week's prompt isn't a prompt to use in production — it's a prompt that helps you build an eval set for the prompts you already have. Paste in any production prompt and a description of what "good" looks like, and get back a starter test set you can drop straight into a tool like Promptfoo.
You are helping me build an evaluation set for a prompt I use in production. I will give you: 1. The prompt itself (the template sent to the model) 2. A description of what a "good" output looks like Generate the following: 1. FIVE TYPICAL INPUTS Realistic examples this prompt will commonly encounter in normal use. 2. FIVE EDGE CASE INPUTS Unusual, ambiguous, conflicting, or adversarial inputs that could plausibly break this prompt. 3. EXPECTED BEHAVIOR For each of the ten inputs above, one sentence describing what a correct response must do (not the full output — just the criteria for "correct"). 4. PREDICTED FAILURE MODES Based on how the prompt is worded, identify 2-3 specific ways it is likely to fail — e.g. ambiguous instructions, missing constraints, undefined behavior for edge cases, or output format drift. Be specific and concrete. Avoid generic test cases — tailor everything to what this exact prompt is asking the model to do. My prompt: [PASTE YOUR PROMPT TEMPLATE HERE] What "good" looks like: [DESCRIBE THE CRITERIA FOR A CORRECT RESPONSE]
Run this once against any prompt you're relying on, turn the output into ten rows in Promptfoo, and you've gone from "looks fine to me" to an actual regression test in under twenty minutes.
New issue every weekday at 7am. Free forever.
No spam. Unsubscribe in one click.