Most model benchmarks are not much use once you try to make a product decision from them.
They were run from somebody else’s account, through somebody else’s provider, from another part of the world, with a prompt you will never send. Then a colourful chart tells you which model “won”.
That is fine if you enjoy charts. It is not enough if you are deciding what should power a chat, an agent, a coding workflow, or a product that has to behave well on Monday morning.
So PAELLADOC now has its own inference benchmark module.
It runs controlled campaigns through the routes you have configured on your Mac and keeps the result local: the workload, the endpoint, the parameters, the environment, what came back, how long it took, and whether it failed.
Not a model leaderboard. A way to see what your own stack is doing.
Why this needed to exist
“Fast” is a surprisingly useless word when you are talking about an AI response.
A model can start replying in a second and take two minutes to finish. Another can leave the user staring at an empty message for half a minute, then stream at a decent pace. A third can be quick until you put 100K tokens in front of it or run ten requests at once.
Those are different problems. Until now, it was too easy to flatten them into a vague feeling: this route seems slow; that one feels good; this provider was flaky last week.
The new module turns that feeling into a campaign you can inspect later.
What a campaign records
You pick a controlled workload and a route. PAELLADOC creates the samples, runs them, and keeps the context around the measurement.
The controlled workloads cover 1K, 10K and 100K text contexts, plus a vision workload. You can run a single request or a parallel scenario. That matters because a model that looks fine in a short, quiet request can behave very differently under a large context or concurrency.
For every run, the module records the things you actually notice:
- when the first token arrived;
- when the first usable answer arrived;
- how quickly output was generated once streaming began;
- when the complete response arrived;
- token usage reported by the provider;
- retries, failures and incomplete samples.
It also records the route behind the number: provider, endpoint surface, account tier, model identity, parameters, client environment and the benchmark protocol. If those details are missing or the model identity is not trustworthy, PAELLADOC can suppress the performance result instead of presenting a tidy number that should not be compared.
That last part is not decoration. It is the difference between measuring a route and making up a league table.
The first campaigns already made the point
I have been using the module to run the first 10K and 100K campaigns. The data is still small and not ready to be turned into grand claims. That is exactly why the module is useful.
One 10K route began showing text in roughly a second. Another took more than thirty seconds before the first visible token. The second route may still be useful for the right work; it is simply not the route I would put behind a chat where a person is waiting.
The 100K parallel runs are even more instructive. Some samples completed, some failed, and some are not terminal yet. The correct output is not a hand-picked speed chart. It is an incomplete campaign with the failures still attached.
That is the behaviour I wanted from the module: tell me when I do not have enough evidence yet.
What it is for
This is not a claim that PAELLADOC can tell you which model is “best”. A latency run does not know whether code is correct, whether a plan is good, whether the model handled a tool safely, or whether its data policy is acceptable.
It does make the next decision less hand-wavy.
If you are building an interactive flow, look at the wait before the first token. If you are generating a long document in the background, look at the full completion time. If you are routing an agent over a large repository, run the workload that resembles that job. If the route is unreliable, do not hide the failures just because the successful samples look quick.
Then take the result into the wider routing decision: quality, cost, privacy, availability and validation still matter. That is the job of model routing. The benchmark module gives that decision layer something better than instinct.
Local by default, exportable when you need it
The campaigns live with the rest of the project data on your machine. They are not training data for a shared public ranking. You can inspect a campaign in the app, export its report as Markdown, or use its structured result elsewhere.
That makes the module useful in two different ways. First, it answers a practical question today: which configured route should I use for this kind of work? Second, it leaves a record of why that choice made sense at the time, so you can rerun it when a provider, model, account or network path changes.
What comes next
The module has just started collecting real campaigns. The next job is not to rush a public leaderboard out of a handful of runs. It is to build enough history to see variance, failures and change over time — then connect that data to actual task quality.
That is where a benchmark becomes useful product infrastructure rather than another page of model opinions.
PAELLADOC now has the place to do it.