\n
\nAn agent tool-calling benchmark where latency matters as much as intelligence.<\/p>\n
Most tool-calling benchmarks (BFCL, \u03c4-bench, ToolSandbox, AppWorld) check whether a model
\nmakes the right calls \u2014 and the world politely waits while it thinks. That’s fine for offline
\ntasks. But if you’re building a voice assistant, a live-ops agent, or anything realtime, you
\ncare about two things at once: does the model do the right thing, and does it do it fast
\nenough? A model that finds the perfect answer after thirty seconds of reasoning is, for you,
\nthe wrong model.
\nKitchen Rush measures both at once, by construction: the time a model spends thinking is
\nconverted into game time that passes before its actions land. While the model deliberates,
\nfood keeps cooking, food burns, and order deadlines slip away. Speed and accuracy aren’t two
\ncharts you squint at \u2014 they’re one score, experienced the way a deployment would experience
\nthem.<\/p>\n
The model plays a chef in an Overcooked-style
\nkitchen. Orders stream in (burgers, soups, ramen\u2026), and the model fulfils them with ordinary
\nnative function calls \u2014 collect, chop, cook, plate, serve \u2014 racing deadlines,
\nburn timers, and a combo bonus for consecutive successful dishes. Three deliberate changes from
\nOvercooked:<\/p>\n
Latency is the game. Every model response first charges its thinking time to the shared
\nworld clock, then its actions execute. (You can chain several calls in one response and pay
\nthe latency once \u2014 decisiveness is rewarded.)
\nNo joystick skills. The chef walks itself to the right station automatically; travel
\ntime is charged inside the action. What’s being tested is choosing the right action
\nsequence under time pressure, not video-game reflexes.
\nFully deterministic. Same seed, same actions, same latencies \u2192 exactly the same episode,
\nevery time, on any machine. Every run can be replayed in a browser viewer and audited.<\/p>\n
Every episode produces a single 0\u2013100 score we call KR (the Kitchen Rush score). It’s
\ngraded on a curve between two fixed anchors: KR 0 means “no better than doing nothing and
\nletting every order expire,” and KR 100 means “matched a scripted reference chef that plays
\nthe same kitchen with zero latency.”
\nA worked example makes it concrete. Say that on one kitchen the do-nothing chef finishes at
\n\u221260 points (every order expired), the zero-latency reference chef finishes at +140,
\nand your model finishes at +40. There are 200 points between the two anchors and your
\nmodel covered 100 of them, so its KR is 50 \u2014 it closed half the gap to the reference.
\nAverage that over many seeded kitchens and you have the leaderboard number
\n(docs\/METHODOLOGY.md has the full formula).<\/p>\n
Here’s the knob that makes Kitchen Rush flexible: every kitchen is generated at a latency
\nbudget B (–latency-budget, in seconds per decision). Think of B as the pace the
\nkitchen is priced for: order deadlines are set so that a chef spending exactly B seconds on
\neach decision can finish every order, with roughly 1.4\u20131.6\u00d7 headroom to spare. Each B gets its
\nown leaderboard \u2014 results at different budgets are never averaged together.
\nFor the mathematically inclined, the pricing is exact:
\ndeadline = arrival + \u2308\u03c3 \u00b7 C(B)\u2309, where C(B) = A + K\u00b7B<\/p>\n
A is the order’s intrinsic cooking\/walking time, K is how many decisions a competent plan
\nneeds, and \u03c3 is the headroom (1.4\u20131.6 by tier). So a model that actually decides in \u2113 seconds
\ngains or loses K\u00b7(B \u2212 \u2113) seconds of breathing room per order. Faster than B? You bank slack
\nand serve while orders are still worth full value. Slower? You eat through the headroom, and
\norders start becoming unfinishable at around \u2113 \u2248 B + (\u03c3\u22121)\u00b7C(B)\/K \u2014 about 3\u20134 s\/decision at
\nB=1 on the current tiers, which is exactly where our calibration sweep shows the reference
\nchef collapsing (docs\/METHODOLOGY.md \u00a72,
\ndocs\/CALIBRATION.md).
\nAnd in plain deployment terms: the model that wins at B=1s is the best pick when every
\ndecision has to land in about a second \u2014 on the benchmark’s reproducible clock that’s a
\nbudget of roughly 65 output tokens per decision, i.e. terse, single-shot tool dispatch \u2014 what a
\nvoice agent needs. B=5s buys about 730 tokens per decision \u2014 enough for a short burst of
\nreasoning, what an interactive assistant can afford. The same model can rank very differently on the
\ntwo boards, and that reordering is precisely what the benchmark is for.<\/p>\n
17 model configurations \u00d7 12 seeds \u00d7 {medium, hard} kitchens \u00d7 two latency budgets \u2014 816
\nepisodes so far. Each chart is one latency budget; bars are mean KR, whiskers are 95%
\nconfidence intervals. The full per-tier table (with costs, reasoning tokens, and serve rates)
\nis at leaderboard\/results\/board.md.<\/p>\n
The left board (B=1s) is the realtime test: the kitchen is priced for one second per
\ndecision, which on the benchmark’s clock buys about 65 output tokens \u2014 terse, single-shot tool
\ndispatch. Winning here means “the model I’d trust to drive a voice agent or a live dashboard.”
\nThe right board (B=5s) prices the same kitchens for five seconds per decision (~730
\ntokens \u2014 room for a short burst of reasoning), what an interactive assistant can afford.
\nRead them side by side \u2014 that contrast is the product. Under tight realtime pressure (B=1s)
\nthe fast no-reasoning models hold the podium: gemini-3.1-flash-lite runs nearly even with
\nclaude-sonnet-4.6 (32 vs 37). Give every decision five seconds instead and the board
\nreorders: gpt-5.4-mini with low reasoning rockets from near-zero to a dead heat with
\nsonnet (44 vs 44) at about a fifth of the cost, while flash-lite drops to half its B=1
\nstanding. The same mini with reasoning fully off scores 0.0 at both budgets \u2014 reasoning it
\ncan’t afford at B=1 is exactly what makes it a frontier-level tool caller at B=5. That’s the
\nlatency tax, made visible. (\u00b7think rows ran with reasoning on at low effort; everything
\nelse with reasoning off \u2014 fast single-shot dispatch is the honest realtime default. One row
\nyou might expect is missing: there is no claude-sonnet-4.6\u00b7think, because Anthropic’s API
\ndoes not allow extended thinking when tool calls are forced, and the harness forces tool
\ncalls \u2014 sonnet competes thinking-off only.)<\/p>\n
The flip, watched live: the same two models from the clip at the top,
\nbut in a kitchen priced at B=5s. Now the mini’s reasoning burst is affordable \u2014 it finishes
\nevery order at 99 raw points (KR 86) while sonnet is still cooking at 40. This is the
\nmini’s best kitchen \u2014 the chart above shows the average, a 44\u201344 tie across all 24 \u2014 but the
\ndirection is real: it wins the medium tier at B=5 outright (59 vs 52). Same models, different
\nlatency budget, different winner: that’s exactly what the two boards measure.<\/p>\n
Two minutes \u2014 run the scripted reference chef locally (no model calls):
\npip install -e . # the core has zero dependencies
\nkitchenrush bench –baseline random –tier easy –seeds 12 –trials 2
\nkitchenrush calibrate –tier easy –latency-budget 1 # see how the reference chef degrades with latency<\/p>\n
# watch a game in the browser (scripted chef):
\nkitchenrush replay –oracle –tier easy –seed 0 # writes ui\/replays\/easy_seed0.json
\ncd ui && python3 -m http.server 8000 # then open http:\/\/localhost:8000
\n# …or race up to 4 models side-by-side on one clock: ?replays=a.json,b.json (see ui\/README.md)
\nTo benchmark a real model, add provider support and your API key:
\npip install -e ‘.(providers)’
\nkitchenrush bench –model anthropic:claude-sonnet-4-6 –tier medium –latency-budget 1
\nAny LiteLLM-routable model works via provider:model. You can also plug in a fully custom
\nclient \u2014 it only needs a name and a generate(system, messages, tools) -> ModelResponse
\nmethod, registered with register_adapter. CLI commands: run, bench, replay, seeds,
\ncalibrate.<\/p>\n
If you use Kitchen Rush in your work, please cite it (machine-readable copy in
\nCITATION.cff):
\n@software{kitchenrush2026,
\n author = {Eledath, Bassim},
\n title = {Kitchen Rush: A Benchmark for Accurate and Fast Tool Calling},
\n url = {https:\/\/github.com\/bassimeledath\/kitchen-rush},
\n year = {2026}
\n}<\/p>\n