llm-batcher / inference proxy

Why I added a cost and latency observatory

A proxy that translates and batches requests is still a black box until it can answer two questions about itself: how fast is it, and what is it costing me? Here is why the average latency lies, why percentiles do not, and how the dollar estimate refuses to fool you.

See it

The average hides the pain

A realistic latency distribution has a long tail. Most requests are fast, a few are slow, and the few are what your users feel. Watch where the mean sits versus p95 and p99.

Mean
0 ms
p50 (median)
0 ms
p95
0 ms
p99
0 ms
Max
0 ms

Notice how the mean barely moves when you add a few very slow requests, while p95 and p99 jump. Reporting only the average would hide a real regression that your slowest users absolutely notice.

The decision

Percentiles over averages, a window over all time

The naive surface

  • Report mean latency and a lifetime request count.
  • The mean is dominated by the bulk and blind to the tail.
  • All time stats drift: a slow startup poisons the number forever.
  • Cost reported as a single figure, silently zero for any model you forgot to price.

The observatory

  • p50, p95, p99, and max over a bounded recent window.
  • Counts split into total, success, and error (a small RED surface).
  • Cost is an explicit estimate from a static table.
  • Unknown models counted as unpriced, never as zero, with an estimate_complete flag.
What you get

One JSON snapshot at /metrics

Off by default. With METRICS_ENABLED=1 the proxy records one sample per request and serves this. While disabled the endpoint returns 404 and nothing is collected.

{
  "requests": { "total": 1287, "success": 1273, "error": 14 },
  "latency":  { "sample_count": 1024, "window_size": 1024,
                "p50_ms": 412.0, "p95_ms": 980.5, "p99_ms": 1644.2, "max_ms": 2210.7 },
  "tokens":   { "input": 902145, "output": 318220, "total": 1220365 },
  "cost": {
    "estimated_cost_usd": 1.998,
    "estimate_complete": false,
    "pricing_basis": "static table, USD per million tokens",
    "unpriced_request_count": 6,
    "unpriced_models": { "some-preview-model": 6 }
  },
  "note": "in-process, single-worker, opt-in estimate"
}
Stated on purpose

What it is, and what it is not

ChoiceWhyThe honest limit
Bounded ring buffer of raw samples Exact percentiles for the recent window, no bucket guesswork. Describes recent traffic, not all time history.
Nearest rank percentile method No interpolation, clean edges: p100 is the max, one sample is itself. One sort per snapshot call (cheap because the window is clamped).
Static pricing table The upstream gives exact token counts, so the only error source is price. Prices change; the table is updated by hand.
Unknown models tracked as unpriced A wrong but confident dollar figure is worse than a right but incomplete one. estimate_complete goes false until you add the price.
In process, single worker Simplest thing that produces the numbers the next steps need. Multiple workers each keep their own counters.
Read more

The ideas behind it

Every chip links to a primary source for the concept.

The Tail at Scale Nearest-rank percentile RED method USE method HdrHistogram t-digest quantiles Prometheus histograms and summaries Anthropic pricing FinOps principles