Long context windows didn't kill retrieval — but they killed most of the reasons teams reach for it first. Here's how to tell which camp you're in.
Last issue, we talked about how context windows quietly grew from 8,000 tokens to 2 million — and how most teams are still architecting around a constraint that no longer exists.
That raises an uncomfortable follow-up question: if the constraint is gone, why does almost every AI product shipping today still come with a full retrieval-augmented generation pipeline bolted to the front of it?
The honest answer, for a lot of teams, is that RAG became the documented best practice sometime in 2023, and nobody has gone back to ask whether the original justification still holds. That's not a criticism — it's just how engineering culture works. We adopt patterns under one set of constraints and keep them long after the constraints change.
Strip away the tooling and RAG exists to solve exactly three problems: (1) your corpus is bigger than any model's context window, (2) passing everything every time is too expensive, or (3) your data changes too often to bake into a prompt. That's it. Everything else — the embedding pipelines, the chunking heuristics, the reranking models — is machinery built to serve those three goals.
Here's the thing: for a large share of products being built right now, none of those three conditions actually apply. A "chat with your documentation" tool where the docs are 400 pages fits comfortably in a single context window with room to spare. An internal tool that answers questions about a 200-page policy manual doesn't need a vector database — it needs someone to paste the manual into the prompt.
Ask yourself one question before reaching for retrieval: "would this just work if I pasted the whole corpus into the prompt?"
If the answer is yes, and the resulting cost and latency are acceptable for your use case — ship that. It is dramatically simpler to build, debug, and maintain. You can always add retrieval later, once you've measured a real problem rather than guessed at a hypothetical one.
If the answer is no — your corpus is genuinely millions of documents, or it updates by the minute, or you're serving thousands of tenants who each need strict data isolation — then RAG isn't overbuilt for you. It's the right tool, and you should keep investing in it.
Complexity isn't free, and it doesn't show up on a balance sheet the way infrastructure costs do. Every chunking decision is a place a relevant passage can get split in half. Every embedding model is a dependency that can drift out of sync with your data. Every reranking step is another component that can silently degrade and nobody notices for a month.
None of this is an argument against RAG. It's an argument for treating it as what it is: a real engineering investment that should be justified by a real, measured problem — not adopted by default because it was the popular pattern the year you started building.
The teams shipping fastest right now are the ones willing to ask "do we actually need this?" about the parts of their stack that have quietly become assumptions.
NotebookLM is Google's research-and-synthesis tool, and the core idea is simple but well executed: you upload your own sources — PDFs, docs, links, transcripts — and it builds a "notebook" that's grounded only in what you gave it. No hallucinated citations from the open web, no mixing your material with whatever the model half-remembers from training.
I used it for a week to get through a stack of research papers and competitor documentation that had been sitting in a "read later" folder for two months. Asking it direct questions across a dozen sources at once — and getting answers with links back to the exact passage — turned a multi-day slog into an afternoon.
The unexpectedly useful part: the "Audio Overview" feature, which generates a podcast-style discussion of your sources. It sounds gimmicky until you're three commutes deep into actually absorbing material you'd otherwise have skimmed and forgotten.
This paper asks a question that pairs uncomfortably well with last issue's thesis: okay, models can accept a million tokens — but do they actually use all of them equally? The researchers tested this directly, placing the answer to a question at different positions within a long context and measuring how often the model found it.
The result was remarkably consistent across model families: performance is highest when the relevant information sits at the very beginning or very end of the context, and drops noticeably when it's buried in the middle — even in models explicitly designed and marketed for long-context use. The effect has a name now: "lost in the middle."
This week's prompt is a structural template — not a question to ask, but a way to shape any prompt that includes a large amount of reference material. It's a direct, practical hedge against the "lost in the middle" effect from the paper above.
=== PRIORITY INSTRUCTIONS (read first) === [State the single most important task in 1-3 sentences. This is the actual job — say it plainly, up front.] === REFERENCE MATERIAL === [Paste your bulk content here: documents, code, transcripts, data, whatever the task requires. Use as much as you need — the model can hold it. Just don't assume it weighs every part of this section equally.] === PRIORITY INSTRUCTIONS (repeated — do this now) === [Restate the exact same task from the top, in the same words. This is not redundant filler — it's a deliberate hedge against the "lost in the middle" effect. It costs a few hundred tokens and meaningfully improves follow-through.]
Use this any time you're sending more than a few thousand tokens of reference material alongside an actual task — long documents, large codebases, lengthy transcripts. The repetition at both ends feels wasteful. It isn't. It's the cheapest available fix for the one real limitation long-context models still have.
New issue every weekday at 7am. Free forever.
No spam. Unsubscribe in one click.