Open-weight models from Meta, Mistral, DeepSeek, and Qwen have closed most of the gap with frontier closed models — and that's quietly rewritten the build-vs-buy calculus for almost every AI startup. Here's the new math, and how to tell which side of it you're on.
For most of the last two years, "open weights vs. closed API" was mostly an ideological argument — interesting at a dinner party, irrelevant in a planning meeting. The quality gap between the best open-weight models and the frontier closed ones was wide enough that, for almost any production use case, the decision basically made itself: call the frontier API, eat the per-token cost, move on with your life.
That gap has narrowed enough that it's now a real decision — not a foregone one. The latest open-weight releases sit within a few points of frontier closed models on most benchmarks that map to typical product work: summarization, extraction, classification, drafting, tool-use for well-scoped tasks. The remaining gap is real, but it's concentrated in the hardest reasoning and most open-ended creative work — not in the bread-and-butter tasks most startups actually ship.
Here's the part that actually changes decisions: at low volume, the model choice barely matters financially. If you're sending a few thousand requests a day, the difference between a closed-API bill and a self-hosted open-weight bill rounds to noise next to your AWS bill, let alone payroll. Open weights don't win on cost at this scale — the ops overhead of running your own inference isn't worth it yet.
At high volume, the math flips hard. Once a feature's token usage becomes a real line item in your COGS — think anything customer-facing that runs on every page load, every support ticket, every document upload — the per-token gap between a frontier closed API and an open-weight model served through a cheap inference provider can be 5-10x. At that scale, a model that's "95% as good for 15% of the cost" isn't a compromise, it's the entire unit economics of the feature.
If you read Issue 004, you already have the tool for this: an eval set. Benchmark leaderboards tell you how a model performs on someone else's distribution of tasks. The only number that matters is how a candidate open-weight model performs on your distribution — your prompts, your edge cases, your definition of "correct."
The practical move is small: take the eval-set generator from Issue 004, run it against your current production prompt, and then run that same eval set against two or three open-weight candidates through a single API (more on that in this issue's Tool Drop). In an afternoon, you'll have a real cost/quality comparison instead of a vibe.
One place open weights still trail: tool-calling reliability on long, multi-step agentic tasks — the kind discussed in Issue 003. Open models have gotten much better at structured outputs and function calling, but on agent loops with five-plus sequential tool calls, closed frontier models still tend to be more consistent about not drifting off-script. If your use case leans heavily on long agent chains, weight that in the eval — it's the dimension most likely to surprise you.
Lean open weights (self-hosted or via a cheap inference provider) when: volume is high enough that token cost shows up in your COGS, the task is narrow and well-defined, and you can tolerate a slightly lower ceiling on the hardest cases.
Lean closed API when: volume is low, the task needs the best available general reasoning or creative quality, or your team has no appetite for managing inference infrastructure.
Lean open weights regardless of volume when: data residency or privacy requirements mean the data can't leave your infrastructure at all — increasingly a compliance decision, not just a cost one.
And there's a middle path that's gotten dramatically cheaper this year: fine-tuning. A small open model fine-tuned on exactly your task can beat a generic frontier model on that task, at a fraction of the inference cost. Why that's now a weekend project instead of a research initiative is this issue's Paper Trail.
OpenRouter sits in front of dozens of models — open-weight and closed, from nearly every major provider — behind a single, OpenAI-compatible API. Instead of integrating separately with each model provider, you point your code at one endpoint and switch models with a one-line config change.
For the build-vs-buy question this issue is about, that's the whole appeal: take the eval set you built from Issue 004's prompt, run it against your current closed-API model and two or three open-weight candidates through OpenRouter, and you get a real side-by-side on quality — plus OpenRouter's dashboard shows you the actual per-token cost difference, not a marketing number.
Why this matters more than it sounds: the friction of "trying" an open-weight model used to be the whole problem — new SDK, new auth, new request format, new error handling. Removing that friction means the decision actually gets tested instead of assumed.
This is an older paper, but it's the quiet reason everything in this issue is true. Fine-tuning a large model used to mean updating every one of its weights — billions of numbers — and saving a full new copy of the model for every task you fine-tuned it for. That's expensive in compute and storage, and it's why fine-tuning stayed a research-lab activity for years.
LoRA's idea: freeze the original model entirely, and instead train a pair of small "adapter" matrices that get added on top of select layers. The number of trainable parameters drops by orders of magnitude — often less than 1% of the original model — while performance on the fine-tuned task lands close to full fine-tuning.
The practical effect, four years later: a fine-tuning run that once required a multi-GPU cluster and days of training now fits on a single rented GPU for an afternoon, and produces an adapter file measured in megabytes rather than a duplicate multi-gigabyte model. That's the entire reason "fine-tune a small open model on your exact task" went from a research initiative to a line item.
This week's prompt turns the framework above into a decision you can actually run on a real feature. Describe what you're building and your constraints, and get back a recommendation — open self-hosted, fine-tuned open, or closed API — plus the validation plan to confirm it before you commit.
You are helping me decide whether to use a closed-API model, a self-hosted open-weight model, or a fine-tuned open-weight model for a specific feature. I will give you: 1. A description of the task/feature 2. Expected monthly volume (requests or tokens) 3. Latency requirements 4. Data privacy/residency constraints 5. My team's ML ops capacity (none / some / dedicated) Based on this, do the following: 1. RECOMMENDATION State which of the three approaches is the best starting point, and why — in plain terms tied to the inputs above. 2. COST SHAPE Describe how cost will scale with volume for the recommended approach vs. the alternatives, so I understand whether the recommendation could change as I grow. 3. RISKS List the 2-3 biggest risks of the recommended approach (e.g. quality ceiling, ops burden, lock-in) and a mitigation for each. 4. VALIDATION PLAN Describe a small eval (5-10 cases) I should run before committing, comparing the recommended approach against my current setup. Be specific and concrete — avoid generic advice, tailor everything to the inputs. Task/feature: [DESCRIBE THE TASK] Expected monthly volume: [ESTIMATE] Latency requirements: [E.G. REAL-TIME, FEW SECONDS OK, BATCH/ASYNC] Data privacy/residency constraints: [NONE, OR DESCRIBE] ML ops capacity: [NONE / SOME / DEDICATED TEAM]
If the recommendation is "test it," the validation plan it gives you plugs straight into Issue 004's eval-set generator and this issue's OpenRouter setup — from question to answer in one afternoon.
New issue every weekday at 7am. Free forever.
No spam. Unsubscribe in one click.