The honest answer is that, on my laptop, I couldn’t tell. A single curl against the proxy returns in the same few hundred milliseconds whether batching is on or off. The before and after I wrote about is real, but it is completely invisible in a demo. You only see it when 200 clients arrive in the same 50 ms. Which means the only way to defend the feature, tune it, or trust it is to benchmark it. That is the whole subject here.

The companion visualization makes the motivation concrete: the thundering herd, then the fix. batching.html on GitHub Pages (open it in a browser and watch the uncoordinated spike turn into controlled waves). A benchmark is just that picture with numbers attached.

A demo proves nothing

Here is the uncomfortable part of shipping infrastructure: the failure mode you built the feature to prevent does not show up in the way you naturally test it.

A demo is one request at a time. One request opens one upstream connection, waits, returns. Batching adds a few milliseconds of deliberate delay, so in a demo the batched path is, if anything, very slightly slower. Every instinct says the feature is useless.

The feature only earns its keep under concurrency, and concurrency is exactly the condition a casual test never reproduces. The plain proxy under 200 simultaneous clients opens roughly 200 upstream connections at once, hits the classic thundering herd, and has no single place to enforce a limit. None of that is observable until you generate the load on purpose. So “it works on my machine” is not a weak claim here, it is a meaningless one. The machine was never asked the question the feature answers.

What you cannot see without measuring

Three things stay hidden right up until you put real load through the proxy, and all three are the things that actually matter in production.

Tail latency. Averages lie under load. The number that wakes people up is p95 and p99, the slow tail that a handful of requests fall into when everyone is competing for sockets, DNS, TLS handshakes, and upstream rate limits at once. Google’s Tail at Scale is the canonical argument that the tail, not the mean, defines the user experience. You cannot average your way to a p99. You have to measure it, and you only get a meaningful one under concurrency.

Backpressure and the value of the cap. The microbatching layer puts a concurrency semaphore in front of the upstream so it never sees more than BATCH_MAX_CONCURRENCY calls in flight. Little’s Law says that in a stable system the in-flight work L = λ × W: arrival rate times latency. If upstream latency W spikes, a fixed cap keeps L bounded instead of letting in-flight work explode. That is a lovely argument on paper. It is also untestable on paper. The only way to show the cap turns an unbounded meltdown into a bounded queue is to drive λ up and watch what happens.

That the trade is real. The microbatching post called this a latency-for-stability trade: pay a few milliseconds of waiting, buy upstream protection and predictable tails. A benchmark is what turns that sentence from a claim into a measurement. Without one, it is just a story I am telling about my own code.

What benchmarking actually buys you

This is the part I care about most, because “run a load test” sounds like a chore and is actually the highest-leverage thing you can do to a system like this.

It tunes the knobs with evidence instead of vibes. Microbatching exposes real dials: BATCH_MAX_WAIT_MS (how long to coalesce), BATCH_MAX_SIZE (window size), BATCH_MAX_CONCURRENCY (in-flight cap). Every one of them is a tradeoff. Bigger wait means more coalescing and more added latency. Bigger concurrency means more upstream parallelism and less protection. There is a knee in each of those curves, and you cannot guess where it is. A concurrency sweep finds it. The defaults I shipped (20 ms, size 16, concurrency 8) are starting points; a benchmark is how they stop being starting points.

It makes the design defensible. “I added admission control” is a sentence anyone can say. “Here is throughput and p99 with batching off, here it is with batching on at concurrency 8, and here is where it stops helping” is a different kind of claim. It is the difference between an opinion and a result, and in an interview it is the part worth defending.

It catches regressions. Once a benchmark produces a number, that number is a contract you can diff across commits. Refactor the dispatch path, rerun, compare. A silent 30% throughput regression from an accidental await in the wrong place is invisible to unit tests (the batcher tests are deterministic and run with no network, on purpose) but obvious to a benchmark.

It keeps you honest about limits. Measuring under load is also how you find the things the feature does not fix. The windows are per process, so each worker has its own, and there is no global queue across workers yet. There is no queue-depth cap, so a sustained flood can still grow the pending list (returning 429/503 past a max_pending is the natural next increment). I know those are the weak spots precisely because that is where a benchmark would push the system over. Numbers tell you where to build next.

What a benchmark for this measures

Concretely, the harness I want puts three axes on the table at once, because in LLM serving they trade against each other and a single number hides the deal:

• Throughput: requests per second the proxy sustains as concurrency climbs.
• Latency: the full distribution, p50/p95/p99, not the average.
• Cost: dollars per request, since the point of a proxy in front of a paid API is partly economic.

The experiment is a concurrency sweep comparing BATCH_ENABLED=0 against BATCH_ENABLED=1: hold the workload fixed, raise the number of concurrent clients, and plot all three curves for each mode. The batched line should hold its tail flatter as concurrency rises, at the cost of a small constant latency add at the low end. That crossover is the entire feature, drawn.

This is the pre-production twin of the work I did in the observability post: that one added a live /metrics surface so the running proxy reports its own p50/p95/p99 and a dollar estimate. Same RED instinct, two settings. Benchmarking is how you learn the shape of the system before it ships; the metrics endpoint is how you keep watching it after. You want both, and they reinforce each other: the benchmark tells you what “normal” looks like, so the production metrics can tell you when you have left it.

What this demonstrates

The reflex to measure before believing. A demo answers “does it run.” A benchmark answers “does it do the thing I built it for, and where does it stop.” Tail-latency thinking, admission control you can prove with Little’s Law instead of assert, evidence-based tuning, regression gates, and the honesty to let the numbers mark your limits. The feature was the easy part. Knowing it works is the job.

Visualization: batching.html · Repo: github.com/NeverTheSame/llm-batcher. Next I wire the harness itself, so the curves in this post stop being a plan and start being a chart.

The honest problem up front: a proxy that translates and batches is still a black box until it can answer two questions about itself, how fast is it and what is it costing me. Both of the things it already does are claims about behavior under load, and until now there was no way to check either one.

I also built a small interactive dashboard of the metrics surface, which makes the shape click faster than the JSON does: observability.html on GitHub Pages (open it in a browser to see the percentiles and the cost breakdown laid out).

You cannot tune what you cannot see

The proxy translates between the OpenAI and Anthropic shapes, and it can group concurrent requests into a short window before fanning them out. Did windowing actually smooth the upstream call rate? Did it add latency, and if so how much, and to which requests? How much am I spending per thousand requests, and on which models? Right now those are all shrugs.

Without measurement, the next two roadmap items, backpressure tuning and a benchmark harness, are guesswork. Observability is the dependency that unblocks them. So this increment is deliberately the cheapest thing that produces the data those steps need, and nothing more.

The metrics stack I rejected, and why

The obvious move is to reach for a real metrics stack: a Prometheus client library, a /metrics endpoint in the text exposition format, histograms with predefined buckets, a scrape config. That is the correct answer for a production fleet, and the wrong answer here, for three reasons.

1. It adds a dependency and a serving format whose value only appears once there is a Prometheus server, a Grafana instance, and a multi-host deployment to aggregate. None of that exists in a single-process learning proxy.
2. Prometheus histograms commit you to fixed buckets up front. For a quick look at “what is my p95 right now,” a bounded ring buffer of raw samples is more honest and easier to reason about than bucket boundaries chosen blind.
3. The interesting engineering here is not “wire up a library.” It is the set of small correctness decisions a metrics layer quietly gets wrong: percentile method, unknown-model cost handling, where latency is measured relative to batching, and how errors are accounted. A hand-rolled registry forces each of those into the open.

So the design is an in-process registry with a JSON snapshot. It is explicitly not durable, not multi-worker, and not a substitute for a real metrics pipeline. It is the measurement you need to make the next tuning decision, and it states its own limits. (This is the same instinct I described in a Staff AI Engineer interview: less model magic, more boring serving engineering.)

What real systems do, in miniature

The shape of this feature is borrowed from patterns that show up everywhere once you start measuring latency-sensitive services.

• Tail latency over averages. The canonical argument is Dean and Barroso, “The Tail at Scale.” The mean latency of a service is nearly useless because a small fraction of slow requests dominates the experience, especially when one user action fans out into many backend calls. Report percentiles, and care most about p95, p99, and the max, not the average.
• Quantile estimation on a stream. Systems that cannot store every sample use streaming quantile sketches (t-digest, HdrHistogram, DDSketch, Prometheus summaries). They trade a little accuracy for bounded memory. A bounded ring buffer is the simplest member of that family: it keeps exact samples for a recent window and is exact for that window by definition.
• Cost as a first-class metric. FinOps practice treats spend like latency, something you attribute per request, per model, per tenant, then watch over time. Token-based pricing makes this tractable because the upstream hands you the exact billable quantity, input and output tokens, in every response.
• RED and USE. Tom Wilkie’s RED method (rate, errors, duration) and Brendan Gregg’s USE method (utilization, saturation, errors) are the standard checklists for a request-serving system. This snapshot is a small RED surface: request rate (counts), errors, and duration (percentiles), plus the cost dimension that LLM serving adds on top.

Three ideas that hold the whole thing up

Percentiles by the nearest-rank method. A percentile answers “what latency is this request faster than p percent of the recent window.” There are several defensible definitions; this uses nearest-rank on the sorted window. For a window of n ascending samples, the rank for percentile p is round(p/100 * (n - 1)), clamped into [0, n - 1]. That makes p100 the max, and makes every percentile of a single sample equal to that sample, with no interpolation and no off-by-one at the edges. The cost is one sort per snapshot call, which is fine because the window is bounded.

A bounded window, not all-time. All-time percentiles drift. A service that was slow at startup and fast since then reports a misleadingly high p99 forever. A bounded window, the last METRICS_LATENCY_WINDOW samples, reports recent behavior, which is what you act on. The tradeoff is that a quiet service shows stale samples until new traffic rolls them out. For a tuning tool, that is the right tradeoff.

Cost is an estimate, and it says so. The upstream reports exact token counts, so the only source of error in the dollar figure is the pricing table, which is static and maintained by hand. The dangerous failure mode is a model that is not in the table silently contributing zero dollars, which makes spend look lower than it is. The registry refuses that: unknown-model usage goes into a separate unpriced bucket, never folded into the priced totals, and an estimate_complete flag flips to false whenever any unpriced traffic exists. A wrong-but-confident number is worse than a right-but-incomplete one.

Walking through the code

The feature is one module, app/metrics.py, plus a few lines of wiring in app/main.py.

Pricing lookup with family matching

Model names arrive with version or date suffixes (claude-3-5-haiku-latest, claude-3-5-haiku-20241022). The pricing table is keyed by family, and the lookup matches the longest table key the model name starts with:

def _price_for(model: str):
    name = (model or "").lower()
    best = None
    for key in PRICING_USD_PER_MTOK:
        if name.startswith(key) and (best is None or len(key) > len(best)):
            best = key
    return PRICING_USD_PER_MTOK[best] if best is not None else None

Longest-prefix matching means a more specific family wins over a shorter one if one is ever added, and unknown models cleanly return None instead of a guess.

The percentile helper

def _percentile(sorted_values, pct):
    n = len(sorted_values)
    if n == 1:
        return sorted_values[0]
    rank = int(round((pct / 100.0) * (n - 1)))
    rank = max(0, min(n - 1, rank))
    return sorted_values[rank]

It assumes a non-empty ascending list. The snapshot handles the empty case by reporting null percentiles and a sample_count of zero, so a freshly started proxy returns a well-formed snapshot instead of dividing by nothing.

The registry

Metrics holds counters, a deque(maxlen=window) of recent latencies, priced token totals, a running cost, and the parallel unpriced accounting. Two record methods cover the two outcomes:

• record_success(model, input_tokens, output_tokens, latency_ms) always bumps the total and success counts and appends the latency. It then either adds to the priced totals and cost, or, for an unknown model, adds to the unpriced bucket. Both paths record latency, because a slow successful call is still a data point even if it is unpriced.
• record_error(latency_ms) bumps the total and error counts and records the latency, but touches no token or cost state, because a failed call before the upstream responds has no usage to bill.

Every mutation and the whole snapshot() copy happen under a threading.Lock. On a single event loop the synchronous record methods cannot interleave, but the lock costs almost nothing and keeps the registry safe under the threaded test client and any future move to worker threads. The window size is clamped on construction (max(1, min(window, 100000))) so a fat-fingered env value can never make the per-snapshot sort expensive.

Wiring it in

main.py gains a METRICS_ENABLED flag (default off) and builds the registry in the existing FastAPI lifespan only when enabled. The chat endpoint is refactored so both the batcher path and the direct path run inside one try/except around a perf_counter span:

start = time.perf_counter()
try:
    if BATCH_ENABLED and _batcher is not None:
        anthropic_json = await _batcher.submit(payload)
    else:
        anthropic_json = await _direct_dispatch(payload)
except Exception:
    if _metrics is not None:
        _metrics.record_error((time.perf_counter() - start) * 1000.0)
    raise
latency_ms = (time.perf_counter() - start) * 1000.0
if _metrics is not None:
    usage = anthropic_json.get("usage", {})
    _metrics.record_success(model, usage.get("input_tokens", 0),
                            usage.get("output_tokens", 0), latency_ms)

This measures end-to-end request latency, the number the caller actually experiences. In batch mode that span includes the admission-window wait, which is the honest thing to report, because that wait is real latency the client paid. It is deliberately not labeled “upstream call latency,” since under batching one upstream call can serve several logical requests, and conflating the two would overstate per-request upstream timing.

The endpoint

@app.get("/metrics")
async def metrics():
    if not METRICS_ENABLED or _metrics is None:
        raise HTTPException(status_code=404, detail="Metrics are not enabled.")
    return _metrics.snapshot()

When the feature is off the route returns 404 rather than an empty object. That keeps the default behavior identical to before the feature existed, and avoids exposing a traffic-and-spend surface on a public proxy unless the operator opts in.

Configuration

METRICS_ENABLED=0           # off by default; 1 enables collection and the endpoint
METRICS_LATENCY_WINDOW=1024 # recent samples kept for percentiles (clamped to 100000)

Testing without a network

tests/test_metrics.py is fully deterministic and never touches the network. It drives the Metrics class directly to assert the math, and exercises the endpoint through the app:

• Percentile edges: a single sample, and the nearest-rank values for a known 1..100 window (p50 is 51, p95 is 95, p99 is 99, p100 is 100).
• Pricing: family and dated-alias resolution, and None for unknown models.
• Cost math: a known model where one million input and one million output tokens produce an exact dollar figure, with estimate_complete true.
• Unknown model: zero priced cost, estimate_complete false, and the usage landing in the unpriced bucket instead of the priced totals.
• Error accounting: errors bump the error count and record latency but no tokens.
• Empty snapshot: null percentiles, zero counts, well-formed shape.
• Window clamping: zero clamps up to one, a billion clamps down to 100000.
• Endpoint: 404 by default, and after reloading the module with the flag set, a mocked round trip produces total: 1, success: 1, the right token total, and a positive estimated cost.

The default suite runs with the feature off, proving the existing realtime and translation tests are untouched. The endpoint test reloads the module with METRICS_ENABLED=1 to prove the enabled path, so both modes are covered.

Limitations, stated on purpose

• In process, single worker. Behind multiple workers each process has its own counters; a real deployment would aggregate them in a scrape target.
• Recent window, not history. Percentiles describe the bounded window, not all-time behavior.
• Cost is an estimate from a static table. Prices change and the table is updated by hand. Unknown models are surfaced, never silently zeroed.
• No durability. A restart resets every counter, which is correct for a tuning tool and wrong for billing. Billing-grade accounting belongs in the provider’s own usage records.

What it demonstrates

The same distributed-systems instincts from the last two posts, now pointed at self-measurement: tail-latency thinking, a streaming-quantile primitive picked for the actual scale, FinOps-style cost attribution, a RED surface, and the judgment to reject the heavyweight metrics stack until it earns its place. Plus the small refusals that keep numbers honest: the unpriced bucket, the 404 when disabled, the clamp on the window.

Visualization: observability.html · Repo: github.com/NeverTheSame/llm-batcher. With the observatory in place, the remaining roadmap items become measurable instead of asserted. Backpressure and concurrency tuning can be judged by their effect on p95 and the error rate, and the benchmark harness can read real throughput, latency, and cost out of /metrics under load. The tail latency stops being a claim and starts being a graph.

The counterintuitive part up front: under load, the fastest thing a proxy can do is sometimes wait. A few milliseconds of deliberate waiting buys you upstream protection, smoother throughput, and predictable tail latency. That trade is the whole feature.

I also built a small interactive visualization of the window filling and flushing, which makes the idea click faster than any paragraph: batching.html on GitHub (open the raw file in a browser to watch the waves).

Before and after: 200 clients in the same 50 ms

The plain proxy does the obvious thing: every POST /v1/chat/completions opens its own connection to the Anthropic Messages API and waits. Perfect for one request at a time. Ugly the moment real concurrency shows up.

Before (no batching). 200 clients call you inside the same 50 ms:

• You open roughly 200 simultaneous upstream connections.
• The upstream sees an uncoordinated spike, the classic thundering herd.
• There is no single place to enforce a concurrency limit, a queue-depth limit, or any fairness policy.
• Your p95 and p99 go unpredictable, because every request fights every other request for sockets, DNS, TLS handshakes, and the upstream rate limit at once.

After (microbatching on). The same 200 requests land in an admission window, get grouped, and go out as a coordinated wave behind a concurrency cap. The upstream never sees more than N calls in flight. The spike becomes a shaped flow. You pay a few milliseconds of latency and you get a system that degrades gracefully instead of melting down.

This is the exact gap between “works in a demo” and “survives production traffic,” and closing it is the kind of work an inference-serving team does every day. That is why it was the right second feature.

What it actually does

When BATCH_ENABLED=1, requests are not dispatched immediately. They drop into an admission window that flushes when either trigger fires first:

1. Size trigger: the window reaches BATCH_MAX_SIZE requests, or
2. Time trigger: BATCH_MAX_WAIT_MS milliseconds pass since the first request entered the window.

The window then dispatches together. Each request is still its own realtime Anthropic call, but the calls leave as a coordinated group on a shared HTTP client, gated by a semaphore so the upstream never sees more than BATCH_MAX_CONCURRENCY in flight.

                 admission window (size OR time)
clients ─▶ [ r1 r2 r3 ... rN ] ─▶ dispatch ─▶ semaphore(N) ─▶ Anthropic
                                                   │
                              at most BATCH_MAX_CONCURRENCY in flight

It is a latency-for-stability trade, and it is opt-in. Off by default, the proxy behaves exactly as before.

The decision I rejected (this is the actual engineering)

The roadmap line said “accumulate concurrent requests into one upstream call window.” Read literally, Anthropic ships something that looks like a perfect match: the Message Batches API, which takes many requests in one call and is 50% cheaper. I did not use it here, on purpose.

The Message Batches API is an offline, asynchronous bulk primitive. You submit a batch and poll for completion, with an SLA of up to 24 hours (often minutes, but the contract is “eventually”). Putting that behind a synchronous, OpenAI-compatible chat endpoint would mean a client’s POST /v1/chat/completions could block for minutes. That is surprising, timeout-prone, and wrong for a realtime API.

So the design splits into two tools for two jobs:

Picking the right primitive instead of the one that happens to share your feature’s name is the judgment call worth defending in an interview. The cheaper API is not the correct API here.

It’s the same idea as a dozen systems you already trust

Microbatching is not exotic. It is one pattern wearing many hats:

• Continuous batching in LLM serving. vLLM and TGI batch tokens across concurrent requests to keep the GPU busy. Same instinct (group concurrent work), different layer (token scheduling vs request admission).
• Nagle’s algorithm in TCP. Briefly buffer small sends to coalesce them into fewer, fuller packets. A time-or-size window, exactly like ours.
• Group commit in databases and logging. Batch many writes into one fsync.
• DataLoader batching. Coalesce many field resolutions in a tick into one backend call.

If you have ever tuned any of those, you already understand this layer. That is the point: it is a small, honest instance of a technique that shows up everywhere in high-throughput systems.

Why a concurrency cap is the right lever (Little’s Law)

Two ideas justify the knobs. Admission control is the window plus the semaphore deciding how much work is allowed downstream at once. Backpressure is having a single dispatch point where you can push back when overwhelmed (today via the cap and timeout; a queue-depth limit returning 429/503 is the next increment).

Little’s Law is why the cap is the correct dial. In a stable system, L = λ × W: average in-flight requests (L) equals arrival rate (λ) times average latency (W). When upstream latency W rises, holding L fixed with a semaphore stops in-flight work from exploding. You trade a little queueing delay for a bounded, predictable system instead of an unbounded meltdown. SRE instinct, pointed at an LLM upstream.

The async footguns I had to dodge

The interesting bugs in this kind of code are all in the asyncio lifecycle. The accumulator holds pending (params, Future) items under an asyncio.Lock, and the snapshot-and-clear is atomic, but the upstream work happens outside the lock so a slow dispatch never blocks new arrivals:

async with self._lock:
    self._pending.append(item)
    if len(self._pending) >= self._max_batch_size:
        batch = self._take_pending_locked()                 # size trigger
    elif self._timer is None:
        self._timer = asyncio.create_task(self._timer_flush())  # arm time trigger

Dispatch is where three guarantees come from a handful of lines: a concurrency cap, failure isolation, and no hangs.

async def run_one(item):
    async with self._sem:                      # concurrency cap
        return await self._dispatch_one(item.params)

results = await asyncio.wait_for(              # per-window timeout
    asyncio.gather(*(run_one(it) for it in batch), return_exceptions=True),
    timeout=self._batch_timeout_s,
)

return_exceptions=True means one request blowing up does not fail its neighbors; each Future gets its own result or its own error. asyncio.wait_for puts a hard ceiling on a window, so a stuck batch fails every still-pending caller instead of hanging forever. Three more traps I had to handle explicitly:

• Timer self-cancellation. The size path cancels the timer, but the timer must never cancel itself. The timer clears its own reference under the lock instead of calling the shared cancel helper, avoiding a race that would drop a window on the floor.
• Task garbage collection. Background dispatch tasks are kept in a set until done, because the event loop will happily GC a task you don’t hold a reference to, mid-flight. A real footgun.
• Caller cancellation. If a client disconnects before its window flushes, submit removes its pending item so the proxy never spends an upstream call on a result nobody is waiting for.

Graceful shutdown (aclose()) cancels the timer and fails any leftover futures, so a shutdown never leaves a request hanging.

Tested with no network and no key

Six deterministic tests run the batcher against an injected async fake, so behavior is fully controllable with no API key and no network:

1. Flush by size groups into exactly one window.
2. Flush by time releases a partial window after the wait.
3. Results route back to the correct caller even when dispatch finishes out of order.
4. A per-item error is isolated; neighbors still succeed.
5. A stuck dispatcher trips the batch timeout and fails every caller (itself bounded by wait_for in the test, so a bug cannot hang CI).
6. Shutdown resolves any still-queued futures instead of hanging.

The original translation tests keep passing, because batching defaults to off.

The knobs

All opt-in, all environment variables:

Tuning intuition: bigger BATCH_MAX_WAIT_MS means more coalescing and more added latency; bigger BATCH_MAX_CONCURRENCY means more upstream parallelism and less protection. These are the same dials a real serving system exposes, and the defaults favor low latency.

Honest limitations

• Per process. Each worker has its own window. This protects the upstream but is not a single global queue across workers. On purpose, and documented.
• No queue-depth limit yet. A sustained flood can still grow the pending list. A max_pending returning 429/503 is the natural next increment.
• N calls, not literally one. This is operational batching (coordinated waves), not a single multiplexed upstream call. That is the correct trade for a realtime endpoint, and I keep the naming precise about it.

What it demonstrates

Distributed-systems instincts pointed at LLM serving: admission control, backpressure, concurrency limiting, failure isolation, timeout discipline, clean async lifecycle management, and the judgment to pick the right primitive over the literally-named one. The job, in miniature.

Visualization: batching.html · Repo: github.com/NeverTheSame/llm-batcher. Next brick is the cost and latency observatory: per-request p50/p95/p99 and a dollar estimate, so the tail latency I keep talking about stops being a claim and starts being a graph.

• Repo: github.com/NeverTheSame/llm-batcher
• Stack: Python 3.12, FastAPI, httpx (async), Pydantic v2, Docker
• Tests: 5 unit tests, mocked upstream, no API key required, all green

It’s deliberately small: a clean pass-through with request/response translation and a real test suite. But it’s the seed of the thing an inference-serving team actually runs in production: adaptive batching, routing, backpressure, and a cost/latency observatory.

Why a proxy at all

I’ve spent years as an SRE keeping distributed systems alive: Kubernetes, Terraform, observability, the on-call pager. I’m deliberately pointing those skills at AI inference, the part of the ML stack that’s least about training and most about the thing I already do: serving traffic reliably, cheaply, and fast at scale. (I said roughly this in a Staff AI Engineer interview, less model magic and more boring engineering.)

The insight that unlocked the project: an LLM inference proxy is just a reverse proxy with interesting backpressure. Connection pooling, request batching, queueing under load, latency budgets, cost telemetry. Those are exactly the problems I’ve solved for databases and microservices. I don’t need to learn ML from scratch; I need to point existing skills at a new kind of upstream. It’s the same “treat generative AI like any other production workload” stance I took on Composer LLM and the Hugging Face model radar, just from the serving side instead of the model-selection side.

Before and after: switching a real app to Claude

Here’s the concrete payoff, the thing I care about most: a real app, before and after. Say you have a small tool built on the official OpenAI Python SDK. It works, it’s in production, and now someone asks: “can we try Claude for this?”

Before (no proxy): to answer that question you rewrite the call against the Anthropic SDK. Watch how many lines that touch the API have to change.

# BEFORE: the app as it ships today, talking to OpenAI
from openai import OpenAI

client = OpenAI()  # reads OPENAI_API_KEY
resp = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are terse."},
        {"role": "user", "content": "Reply with exactly: pong"},
    ],
    max_tokens=16,
)
print(resp.choices[0].message.content)

# BEFORE: the rewrite you need to run that same logic on Claude
from anthropic import Anthropic

client = Anthropic()  # different client, reads ANTHROPIC_API_KEY
resp = client.messages.create(
    model="claude-3-5-haiku-latest",
    system="You are terse.",                  # system prompt leaves the messages array
    messages=[
        {"role": "user", "content": "Reply with exactly: pong"},
    ],
    max_tokens=16,                            # no longer optional
)
print(resp.content[0].text)                   # response is a list of typed blocks

Every line that touches the API changed: the client class, the system-prompt placement, the now-mandatory max_tokens, and the response parsing. Multiply that across a codebase with retries, streaming, function-calling, logging, and tests, and “let’s just try Claude” quietly becomes a multi-day migration.

After (with llm-batcher): the app doesn’t change at all. You point the OpenAI client at the proxy and ask for a Claude model. That’s the entire diff.

# AFTER: same OpenAI SDK, same message shape, same parsing
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1")  # the only line that changed
resp = client.chat.completions.create(
    model="claude-3-5-haiku-latest",   # a Claude model id
    messages=[
        {"role": "system", "content": "You are terse."},
        {"role": "user", "content": "Reply with exactly: pong"},
    ],
    max_tokens=16,
)
print(resp.choices[0].message.content)

Or change nothing in code at all, because the OpenAI SDK already honors an env var:

export OPENAI_BASE_URL="http://localhost:8000/v1"

The proxy absorbs every contract difference, so the application code stays exactly as it was:

That’s the whole idea: meet the tool where it already is. The interesting engineering is in how the proxy absorbs those differences, which is the rest of this post.

What it does

┌────────────────┐   OpenAI-shaped    ┌───────────────┐   Anthropic-shaped   ┌──────────────────┐
│ OpenAI client  │ ─ POST /v1/chat/ ─▶│  llm-batcher  │ ─ POST /v1/messages ▶│  Anthropic API   │
│ (SDK, curl,    │   completions      │  (translates  │                      │  (Claude models) │
│  LangChain…)   │◀─ chat.completion ─│   both ways)  │◀─ messages response ─│                  │
└────────────────┘                    └───────────────┘                      └──────────────────┘

Concretely, the proxy:

1. Accepts an OpenAI POST /v1/chat/completions request, the de-facto shape nearly every LLM tool already emits.
2. Translates it to Anthropic’s Messages format (the two contracts differ in subtle, important ways; more below).
3. Forwards it to https://api.anthropic.com/v1/messages over async httpx, with a timeout and proper error propagation.
4. Translates Claude’s response back into the OpenAI chat.completion shape, including usage token counts and a mapped finish_reason, so the caller never knows it wasn’t talking to OpenAI.
5. Exposes GET /health for liveness checks (the SRE in me refuses to ship a service without one).

The interesting part: the two APIs are almost the same

This is the kind of detail that looks trivial until you wire it up. OpenAI and Anthropic both do “chat,” but the contracts diverge in ways that will bite you.

1. The system prompt lives in a different place

OpenAI carries the system prompt as a message with role: "system" inside the messages array. Anthropic does not allow a system role inside messages. The system prompt is a top-level field. So the translator has to lift every system message out of the array and join them into the top-level system field:

def _to_anthropic_payload(req: ChatCompletionRequest) -> dict[str, Any]:
    system_parts: list[str] = []
    messages: list[dict[str, str]] = []
    for m in req.messages:
        if m.role == "system":
            system_parts.append(m.content)
        else:
            role = "assistant" if m.role == "assistant" else "user"
            messages.append({"role": role, "content": m.content})

    if not messages:
        raise HTTPException(status_code=400,
                            detail="At least one non-system message is required.")

    payload = {
        "model": req.model or DEFAULT_MODEL,
        "max_tokens": req.max_tokens or DEFAULT_MAX_TOKENS,
        "messages": messages,
    }
    if system_parts:
        payload["system"] = "\n\n".join(system_parts)
    return payload

2. max_tokens is optional in OpenAI but required in Anthropic

Forward an OpenAI request that omits max_tokens and Anthropic rejects it. The proxy defaults it (DEFAULT_MAX_TOKENS, env-configurable) so callers don’t have to care.

3. The response envelope is shaped differently

Anthropic returns content as a list of typed blocks with its own stop-reason and usage names. OpenAI clients expect choices[].message.content, a finish_reason, and usage keyed as prompt_tokens / completion_tokens / total_tokens. The translator flattens the blocks, maps the stop reason, and renames the usage fields:

def _to_openai_response(anthropic_json, model):
    text = "".join(b.get("text", "")
                   for b in anthropic_json.get("content", [])
                   if b.get("type") == "text")
    stop = anthropic_json.get("stop_reason")
    finish_reason = "stop" if stop in ("end_turn", "stop_sequence") else "length"
    usage = anthropic_json.get("usage", {})
    pt, ct = usage.get("input_tokens", 0), usage.get("output_tokens", 0)
    return {
        "id": "chatcmpl-" + uuid.uuid4().hex,
        "object": "chat.completion",
        "created": int(time.time()),
        "model": model,
        "choices": [{"index": 0,
                     "message": {"role": "assistant", "content": text},
                     "finish_reason": finish_reason}],
        "usage": {"prompt_tokens": pt, "completion_tokens": ct,
                  "total_tokens": pt + ct},
    }

4. Auth and versioning headers differ

OpenAI uses Authorization: Bearer <key>. Anthropic uses a custom header pair, and forgetting anthropic-version is a classic first-time 400:

headers = {
    "x-api-key": api_key,
    "anthropic-version": "2023-06-01",
    "content-type": "application/json",
}

Pinning the version in one place means I never chase that bug across call sites.

Engineering choices (and the SRE reasoning behind them)

Async all the way down (httpx.AsyncClient). A proxy’s whole job is to wait on an upstream. Blocking I/O would cap concurrency at the worker count; async lets one process hold thousands of in-flight requests cheaply, which matters enormously once batching enters the picture, because batching is literally “hold requests and wait.”

Timeouts and explicit error mapping. Every upstream call has a timeout. Network failures become 502 Bad Gateway; a missing key becomes a clear 500 with a human message; Anthropic’s own errors are propagated with their status code. No silent hangs, no leaked stack traces. That’s the difference between “works on my laptop” and “survives a bad afternoon in prod.”

Config via environment, secrets via .env (git-ignored). ANTHROPIC_API_KEY, DEFAULT_MODEL, DEFAULT_MAX_TOKENS, and REQUEST_TIMEOUT_S are all env-driven. The repo ships a .env.example and a .gitignore that keeps real keys out of version control, verified before I made the repo public.

Pydantic v2 request models. Incoming requests validate against a typed schema, so malformed input fails fast with a 422 instead of exploding three layers deep.

Testing without spending a cent (or leaking a key)

I wanted a suite that runs in CI with no API key and no network, because tests that hit the real API are slow, flaky, and cost money, and a key in CI is a key that can leak. So the upstream Anthropic call is mocked and the deterministic translation logic is tested directly. Five tests cover:

1. System-message lifting into the top-level system field
2. Rejecting a request with no non-system message
3. Response-shape translation (content flattening, finish_reason, token math)
4. A full mocked round-trip through the FastAPI endpoint
5. The /health endpoint

$ pytest -q
.....                                                    [100%]
5 passed in 1.53s

There’s also a tests/smoke.sh for a live round-trip once you drop in a real key, but it’s never required for the suite to pass.

Try it yourself

git clone https://github.com/NeverTheSame/llm-batcher
cd llm-batcher

python3 -m venv venv
./venv/bin/pip install -r requirements.txt

cp .env.example .env          # then paste your ANTHROPIC_API_KEY
./venv/bin/uvicorn app.main:app --reload --port 8000

Now talk to Claude using an OpenAI-shaped request:

curl -sS -X POST http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-3-5-haiku-latest",
    "messages": [
      {"role": "system", "content": "You are terse."},
      {"role": "user", "content": "Reply with exactly: pong"}
    ],
    "max_tokens": 16
  }'

Or run it in a container:

docker compose up --build

Where it’s going

Today it’s a pass-through. The name, llm-batcher, is a promise about the next bricks: request batching (accumulate concurrent requests into one upstream window), a cost & latency observatory (per-request p50/p95/p99 and a dollar estimate), backpressure and concurrency caps (graceful degradation under load), and a benchmark harness to put throughput-vs-latency-vs-cost numbers on the table.

The point isn’t to build a production gateway; those exist. The point is to demonstrate, in public, that I think about LLM serving the way an inference engineer does: in terms of tail latency, fleet efficiency, and dollars per million tokens, not model accuracy.

Repo: github.com/NeverTheSame/llm-batcher. Follow along to watch a reverse proxy turn into an inference-serving primitive.

What it actually is: the first piece of AI tooling I’ve used that runs evals on the engineer instead of the model. No data leaves your machine. The rules — 45 of them, across prompt quality, session hygiene, code review, tool mastery, and context management — run locally against your VS Code session logs and tell you, with receipts, how good (or sloppy) your agentic-coding practice has been this week.

I’ve been quietly arguing some version of this for a while (Staff AI Engineer interview → “the model is a junior; the senior is whoever defines the eval”). AI Engineer Coach takes that to its logical conclusion: the senior is also whoever measures themselves.

Three things that surprised me

1. It scores your AGENTS.md

There’s an “instruction-file audit” pass. If your AGENTS.md is vague, missing, or contradicts itself, the coach will tell you. This is the most satisfying moment of dogfooding I’ve had this year, because back in August I wrote Agents.md ate my README arguing that this file was about to become a standard. It is — and now Microsoft is shipping a linter for it. Full circle 🔁.

The check is dumb-simple: does the file exist, does it cover setup / test / style / branch policy, is it short enough that an agent will actually read it. That’s it. But “dumb-simple thing the team kept forgetting” is the whole job description of half the SRE work I’ve ever done.

2. Anti-patterns are mostly about you, not the agent

The list reads like the worst quarterly review you’ll never have to write about yourself:

• prompting the agent without ever pasting the failure
• never reading what it changed before accepting
• 12 sessions on the same problem with no rollback
• letting context drift past the model’s window because you can’t be bothered to start a new session

This is what eval culture should have been doing for AI users all along. The model providers built evals for their own training loops. We — the people who type into the chat — built nothing for ourselves. The coach finally does.

3. “Skill Finder” turns repeated prompts into reusable skills

If you keep typing the same prompt variants (“turn this into a runbook”, “explain like I’m an on-call engineer at 2am”), it surfaces the pattern and offers to extract it as a saved skill. This is the same instinct behind the Forward Deployed Engineering workflow: repeated bespoke work → productized tool. The coach is just doing it for your own prompts.

What it’s not

It’s not telemetry going home. The repo is MIT, the analysis is local, and the only “share” surface is a screenshot generator if you actually want to post your stats. I ran it for three days against my Copilot CLI sessions and the scariest finding was the number of sessions where I accepted a change without reading it. Not the model’s fault. Mine.

Why this matters more than another agent benchmark

The industry has been measuring models for two years and almost nobody for practice. SWE-bench, MMLU, HumanEval — fine. But none of those tell me whether I, personally, am writing prompts that produce working code or whether my AGENTS.md is doing its job. AI Engineer Coach is the first widely-available answer to that question, and the fact that it ships as a VS Code extension you can code --install-extension in under a minute is probably its most underrated feature.

It also closes a loop I keep talking about: agentic engineering needs the same observability discipline as production systems. Logs, dashboards, anti-pattern alerts, weekly trends. We built all of that for our services. We had built basically none of it for our own AI coding practice. Now there’s at least a starting point.

Try it: clone, npm ci && npm run package, install the VSIX, point it at your VS Code session folder, and look at your own anti-pattern board. Be prepared to flinch.

The numbers, if you’re shopping:

• Base: $174k–$265k
• TC at top AI startups: $400k–$630k
• At Palantir, FDEs are reportedly ~50% of the technical workforce

Big if true. So what is it, actually?

It’s not a sales engineer with a laptop

An FDE is a software engineer who ships production code inside the customer’s environment. Not a slide deck. Not a POC. A running system, wired into legacy IT, weird auth, weirder data, and a compliance team that has Opinions.

The mental model I like: a startup CTO embedded in someone else’s company for 6 months. You own the outcome, you write the code, you eat the on-call pager when the LLM agent decides Tuesday is a great day to hallucinate invoice numbers.

It’s adjacent to — but distinctly not — these roles:

• Sales engineer — demos and pre-sales. FDE shows up after the contract.
• Solutions architect — diagrams and guidance. FDE writes the code that backs the diagram.
• Consultant — bills hours, leaves a deck. FDE leaves a deployed service.
• Customer success — adoption metrics and QBRs. FDE owns whether the thing works in prod.

Why now

Because the “95% of AI pilots produce no measurable ROI” stat finally made it into board decks.

Every enterprise has bought GPT seats, signed an Anthropic contract, slapped a copilot on something. Almost none of them have a working deployment. The bottleneck was never the model — it was:

• 17-year-old Oracle databases with column names like CUST_TYP_3
• A VPN that requires a hardware token issued by a vendor that went bankrupt in 2019
• A security review process that thinks “SaaS” means “sus”
• A data team that — fairly — won’t hand over PII to a model whose system prompt is in a Notion page

You don’t fix that with a better model. You fix it with an engineer who will sit in a Teams call at 7am and rewrite the integration until it works. That’s the FDE.

What the day actually looks like

From talking to a few FDEs at AI labs and reading the postings:

1. Discovery week: shadow the customer’s analyst / ops person. Find the spreadsheet that runs the business. (There is always a spreadsheet that runs the business.)
2. Prototype week: stand up an agent / RAG pipeline / fine-tune that replaces the worst 30% of the spreadsheet. Show the analyst. They cry happy tears or tell you the prototype is wrong in 14 specific ways. Both are wins.
3. Hardening: auth, audit logs, eval harness, rollback plan, on-call rotation. This is the part that separates FDEs from “prompt engineers with consulting hours.”
4. Handoff or expand: either you train the customer’s team and leave, or — more often lately — the customer asks you to do the next workflow, and the next one, until you’ve quietly become their AI platform team.

The skill stack

• Python + TypeScript — non-negotiable
• LLM plumbing — RAG, agent loops, tool calling, evals (Braintrust / LangSmith / homegrown)
• Cloud + IaC — at least one of AWS/GCP/Azure, plus Terraform or Pulumi
• Data — SQL, dbt, Snowflake/BigQuery/Databricks, enough Spark to be dangerous
• Security-ish — SSO, IAM, VPC peering, data residency, “what does SOC 2 actually require”
• People skills — you will get yelled at by a VP who doesn’t understand why their PDF doesn’t parse. Don’t yell back.

The SRE/DevOps crowd is honestly well-positioned here. You already know that prod is where dreams go to die. The only new muscle is doing it inside someone else’s prod.

Is it durable, or just a 2026 bubble role?

My bet: durable, but the title will fragment.

In two years “FDE” will split into:

• Deployment engineers at the AI labs (closer to today’s FDE)
• AI platform engineers inside enterprises (the people the labs hand off to)
• AI-native consulting (EY, Accenture, the new wave of boutique shops)

The work doesn’t go away — enterprise integration never goes away. The label might.

If you’re considering jumping

Three honest filters:

1. Do you like customers? Real ones. With problems. Who interrupt you.
2. Are you willing to write boring glue code in exchange for outsized impact and comp?
3. Can you context-switch between a board-level outcome conversation and a 3am kubectl logs session?

If yes to all three: it’s probably the best comp-per-stress-unit role in tech right now, and the moat (enterprise reality is messy) is structural.

If no to any: stay platform-side. Build the tools FDEs use. That’s also a good job.

— k

Posting this mostly for me — a sanity-check of the answers I gave in a recent Staff AI Engineer loop. Also because writing it down is cheaper than therapy 💸. Names of the hiring company and my current employer are off-limits, so I’ll keep the specifics generic, but the technical substance is exactly what I said in the room.

Background they asked about

My path is one I’ve seen more and more of: SRE → DevOps → AI-facing engineering, with the operational instincts still intact (read: I still twitch at the word “deploy on Friday”). Before AI, I did system architecture and a long stint in customer/technical support — which I told them, with a straight face, was the single most useful background for building anything customer-facing on top of LLMs.

“You stop reaching for clever model tricks and start asking ‘what does the on-call human actually need at 2am?’” 🌙

I’ve been shipping AI features since the GPT-3 era — early on, building an abstraction layer over the Azure OpenAI APIs so the rest of the org could stop copy-pasting requests.post snippets. More recently, an automated incident-quality assessment pipeline that other teams ended up adopting, which is honestly the cleanest signal that a tool is working: people steal it.

The system I walked them through

My current charter: an incident-triage automation system for a security product line, processing multiple customer-reported incidents per day. The inputs are messy — missing logs, half-finished troubleshooting notes, screenshots without context, and the occasional “it’s broken, plz fix 🙏”. My job is to turn that into something a human responder can act on in seconds.

The system is more boring than the conference talks make it sound, which is the point. Boring in production is a compliment. Exciting in production means someone’s getting paged.

1. Quality assessment. Before anything else, an LLM call grades the incoming incident against the backend data we already have — does the customer’s description actually match what telemetry shows? Low-quality inputs get bounced back for clarification instead of poisoning the rest of the pipeline. Garbage in, very confidently-worded garbage out.

2. Retrieval. A RAG layer over ~1,600 internal articles plus the public product documentation, indexed in a vector database with semantic matching. Nothing exotic — the work was in chunking and threshold tuning, not algorithm choice. Sorry, no secret sauce 🥲.

3. Historical context. KQL (Kusto) queries pull the five most recent incidents with similar symptoms. When they asked which retrieved context moves the needle most, I said historical incidents — higher than docs, every time. The docs tell you what should happen. Past incidents tell you what actually happens.

4. Synthesis. The model drafts a triage summary with citations back into the retrieved set. Citations are mandatory; uncited answers are dropped. No vibes-based answers, no matter how confident the model sounds.

RAG — what actually moved the metric

This was the section where I expected the deepest follow-ups, and got them. My short version:

“Most of the wins came from things you’d find in a 2023 blog post. The mistake is thinking you’ve already done them.”

• Precision@k as the primary metric, not recall. For triage, missing a relevant article is annoying; surfacing an irrelevant one is poison.
• Recursive / semantic chunking consistently beat fixed-window chunking.
• Relevance threshold of 0.6 before a chunk is allowed into the answer set. Below that, drop it. Counter-intuitively, returning less context produced better answers — the model is happier with a small, clean tray than a giant buffet.
• LLM-as-judge as a second pass — a smaller model scores the synthesized answer against the retrieved chunks. Disagreement triggers a re-run with tighter retrieval.
• Language-specific datasets. English and German incidents get separate weights. The temptation is to treat language as a translation problem. It isn’t — the technical vocabulary, log formats, and even the way customers complain are different.

Evaluation

The interviewer leaned in here, which I took as a good sign.

• Offline-only evals, run in GitHub Actions on every PR. No live A/B in production for the triage path; the cost of a wrong answer is too high.
• 50–100 incidents manually categorized with SMEs as the golden set. We re-do this every quarter — the distribution of incident types drifts faster than you’d think.
• LangFuse for dataset management and trace inspection. I admitted I had previously tried to bend Grafana into doing this and gave up. Grafana is the wrong shape for LLM calls — you’re not looking at metrics, you’re looking at a conversation.
• A separate eval suite per feature — quality assessment, retrieval, synthesis, and the judge each get their own.

Safety, policy, and the human in the loop

The team services some VIP customers where any automated action needs human approval. The implementation I described and would defend in any review:

• Approval requests routed via Teams, with an escalation chain — primary owner, then a colleague, then mobile phones via the on-call system if neither responds in N minutes. The approval is a feature, not friction — but it has to feel fast or people will route around it.
• LangGraph for policy enforcement. Every tool call passes through a policy node that checks:
  • tool name against an allow-list,
  • argument validation (types, ranges, regex on resource IDs),
  • role-based access for the calling agent,
  • a risk level on the tool that maps to required approval.
• Read-only database credentials for any code path that touches sensitive data. “If your agent has DELETE, it will eventually DELETE.” (Frame this and hang it above your desk.)
• Prompt-injection defence: untrusted text (customer descriptions, retrieved docs) is wrapped in clearly delimited blocks, and the system prompt instructs the model to treat anything inside those blocks as data, not instructions. Belt and suspenders, but it has caught real attempts.

Memory & context engineering

This is where I leaned hardest on the SRE background, and I think it landed.

“State is durable. Context is what the model needs right now. Stop mixing them.”

• State vs context split. State lives in a real store and survives across turns. Context is rebuilt every turn and contains only what the next step needs.
• Compression triggers at ~40% of the context window. I explicitly do not wait for the window to fill — by then, performance has already degraded.
• Memory hierarchy:
  • Cold storage — immutable raw logs. Source of truth, never queried by the model directly.
  • Structured memory — extracted facts (entities, decisions, customer preferences) stored as JSON. Queried deterministically.
  • Compressed summaries — what actually goes into the prompt.
• Critical-information tagging before any compression pass. User preferences, explicit decisions, constraints, and tool outputs are tagged high-importance and survive every compression round verbatim. Tool outputs are almost always more important than the model thinks they are.

Skills, MCP, and the capability registry

I’m bearish on giving an agent unrestricted MCP access and I said so out loud. The setup I described:

• Capability registry — every tool has an explicit contract: name, permission level, side-effect class (read / mutate / external), and whether it’s exposed publicly (as a slash command) or only callable internally.
• Dynamic skill selection per conversation. Pipeline:
  1. Pull candidate skills from the registry filtered by the user’s role.
  2. Rank by relevance to the current user message and the conversation’s accumulated context.
  3. Cap at 3–5 MCPs per conversation. More than that and the model’s tool-choice accuracy collapses — we measured it. Apparently agents, like the rest of us, get decision fatigue at the buffet.
• Examples.md files per skill that the model is shown verbatim. Deterministic invocation patterns beat clever tool descriptions every time.
• Hard RBAC enforcement happens outside the LLM, in the policy layer — the model is never trusted to gate its own access.

Observability & infrastructure

• LangFuse is the standard for LLM observability on my team. Grafana stays for the rest of the stack.
• Everything is logged and alerted. Every tool invocation, every retrieval, every policy decision, every approval response. If it isn’t logged, it didn’t happen — and if it didn’t happen, you can’t reproduce the bug that just paged you.
• Strict separation of production and evaluation environments. Different credentials, different data stores, different model deployments.
• Mock tools for CI/CD. The eval pipeline runs against mocks so it can exercise the agent’s reasoning without making real API calls or mutations.
• Sandbox / shadow environments for testing new policies or skills against real traffic patterns without affecting production.
• Dry-run mode is a first-class feature, not a debug flag. Any mutating tool can be run with dry_run=true and the agent’s plan is logged but not executed.

What I’d tell past-me before the loop

A few patterns kept showing up in my own answers that I want to plant flags on:

1. The metric is precision, not recall. This applies to retrieval, to tool selection, to approval workflows. Returning less, but better, beats returning more.
2. State and context are different things. Treating them as one is the most common mistake I see in agent code reviews — including my own old ones.
3. The policy layer is not part of the model. It runs around the model and is allowed to refuse. The model is never the security boundary.
4. Offline evals with a small, hand-curated golden set, refreshed quarterly. Glamour-free, the only thing that actually moved the metric.
5. Operational instincts beat ML instincts for this class of system. If you’re hiring, look at SRE / support / platform résumés.

The verdict

I got the email a couple of days later: invited to the next round 🎉. So either the above answers were directionally right or I caught the interviewer on a generous day — I’ll take either. Next stage is a system-design round, and I suspect “design the incident-triage system without the benefit of having already built it” is going to be the prompt. Updates to follow.

If you’re prepping for a Staff-level AI engineering loop yourself, the section headers above are basically the question list. You’re welcome 🎁.

The AI Infrastructure Leaders

NVIDIA: The GPU Empire

NVIDIA continues to dominate the AI hardware space, with its stock performance closely tied to enterprise AI adoption cycles. Recent developments show:

• Blackwell Architecture Adoption: The Blackwell GPU platform has become the de facto standard for large-scale AI training, driving consistent revenue growth. Data center revenue now represents over 85% of total revenue.
• Enterprise AI Partnerships: Major cloud providers (AWS, Azure, GCP) are committing to multi-year Blackwell deployments, creating predictable revenue streams.
• China Market Challenges: Export restrictions continue to impact revenue, but the company has pivoted to focus on compliant chips and software solutions.

Key Takeaway for DevOps: GPU availability and pricing directly impact your AI infrastructure costs. Budget planning should account for Blackwell-class hardware and potential supply constraints.

Microsoft: The OpenAI Bet Pays Off

Microsoft’s strategic partnership with OpenAI has positioned it as a leader in enterprise AI adoption:

• Azure AI Services Growth: Revenue from Azure AI and OpenAI services has grown 150% YoY, making it one of Microsoft’s fastest-growing segments.
• Copilot Integration: The widespread adoption of GitHub Copilot and Microsoft 365 Copilot has created a sticky revenue stream that scales with developer productivity.
• Infrastructure Investment: Microsoft is investing heavily in data center expansion to support AI workloads, with capex reaching record levels.

Key Takeaway for DevOps: Microsoft’s AI infrastructure investments mean better tooling and integration for Azure-native deployments. Consider Azure AI services for enterprise workloads requiring compliance and security guarantees.

Alphabet/Google: The Gemini Push

Google’s response to ChatGPT has been aggressive, with Gemini models driving cloud revenue:

• Cloud AI Revenue: Google Cloud’s AI services revenue has doubled, driven by Gemini API adoption and Vertex AI platform growth.
• Open Source Strategy: The release of Gemma models and open-source AI frameworks has created developer mindshare, translating to enterprise adoption.
• Search AI Integration: Google Search’s AI Overview feature has driven engagement metrics, though initial rollout challenges impacted stock performance.

Key Takeaway for DevOps: Google’s open-source AI models (Gemma) provide alternatives to proprietary APIs, enabling on-premises deployments and reducing vendor lock-in.

Amazon: AWS AI Services Scale

Amazon’s Bedrock platform and SageMaker have become critical infrastructure for AI workloads:

• Bedrock Adoption: AWS Bedrock has seen rapid enterprise adoption, with revenue growing 200% YoY as companies migrate from direct API calls to managed services.
• Inferentia and Trainium: Custom AI chips are reducing AWS’s dependency on NVIDIA while offering cost advantages to customers.
• Enterprise AI Contracts: Large multi-year AI infrastructure contracts are becoming common, providing revenue predictability.

Key Takeaway for DevOps: AWS’s custom AI chips (Inferentia, Trainium) can reduce inference costs by 40-60% compared to GPU-based solutions. Evaluate these for production workloads.

Emerging AI-First Companies

Palantir: Enterprise AI Platform

Palantir’s shift to AI-powered analytics platforms has driven significant stock appreciation:

• AIP Platform Growth: The Artificial Intelligence Platform (AIP) has become Palantir’s fastest-growing product, with enterprise customers adopting it for operational AI.
• Government Contracts: Continued strong performance in government contracts provides revenue stability.
• Commercial Expansion: Commercial revenue growth has accelerated as enterprises seek AI-powered decision-making tools.

Key Takeaway for DevOps: Palantir’s AIP demonstrates the value of AI platforms that integrate with existing enterprise infrastructure. Consider similar platform approaches for internal AI deployments.

C3.ai: Enterprise AI Applications

C3.ai focuses on industry-specific AI applications:

• SaaS Transition: The shift to SaaS delivery models has improved margins and customer acquisition.
• Industry Verticals: Strong performance in energy, manufacturing, and financial services verticals.
• Platform Approach: C3.ai’s platform enables rapid deployment of industry-specific AI applications.

Key Takeaway for DevOps: Industry-specific AI platforms can accelerate time-to-value compared to building custom solutions. Evaluate vertical-specific platforms before building from scratch.

Snowflake: Data + AI Convergence

Snowflake’s integration of AI capabilities into its data platform has driven growth:

• AI/ML Features: Native AI/ML capabilities within Snowflake reduce data movement and improve performance.
• Cortex AI: Snowflake Cortex provides LLM capabilities directly on data, eliminating the need for separate AI infrastructure.
• Enterprise Adoption: Large enterprises are consolidating data and AI workloads on Snowflake’s platform.

Key Takeaway for DevOps: Unified data and AI platforms reduce infrastructure complexity. Consider platforms that combine data warehousing with AI capabilities.

Market Trends and Implications

Infrastructure Investment Cycle

The AI infrastructure investment cycle is entering a new phase:

• Training to Inference Shift: As models mature, spending is shifting from training infrastructure to inference infrastructure.
• Edge AI Growth: Edge AI deployments are growing, driven by latency requirements and data privacy concerns.
• Open Source Impact: Open-source models are reducing the cost of AI adoption, impacting proprietary AI service providers.

Enterprise AI Adoption Patterns

Stock performance correlates with enterprise AI adoption:

• Platform Plays Outperform: Companies offering AI platforms (Microsoft, Google, AWS) outperform point solutions.
• Vertical Integration Matters: Companies that integrate AI into existing products (Microsoft 365, Google Workspace) show stronger growth.
• Developer Tools: AI-powered developer tools (GitHub Copilot, Cursor) are creating new revenue streams.

Regulatory and Geopolitical Factors

• Export Restrictions: US restrictions on AI chip exports to China impact NVIDIA and other hardware providers.
• AI Regulation: EU AI Act and similar regulations create compliance requirements that favor established providers.
• Data Sovereignty: Requirements for on-premises AI deployments benefit companies offering hybrid solutions.

Practical Implications for DevOps Teams

Infrastructure Planning

1. GPU Availability: Monitor GPU supply chains and pricing. Consider alternatives (AWS Inferentia, Google TPU) for cost optimization.
2. Multi-Cloud Strategy: Diversify AI workloads across providers to avoid vendor lock-in and optimize costs.
3. Open Source Models: Evaluate open-source models (Llama, Gemma, Mistral) for on-premises deployments to reduce API costs.

Cost Management

1. Inference Optimization: Focus on inference cost optimization as training costs stabilize.
2. Model Selection: Choose models based on cost-performance tradeoffs, not just capability.
3. Reserved Capacity: Consider reserved capacity commitments for predictable workloads to reduce costs.

Platform Selection

1. Integration Requirements: Choose AI platforms that integrate with existing infrastructure and tooling.
2. Compliance: Ensure AI platforms meet regulatory requirements (GDPR, HIPAA, SOC 2).
3. Vendor Lock-in: Prefer platforms that support open standards and allow model portability.

Looking Ahead

The AI stock market is maturing, with clear winners emerging in infrastructure, platforms, and applications. For DevOps teams, understanding these trends helps with:

• Technology Selection: Choosing AI infrastructure and platforms aligned with market leaders.
• Cost Planning: Anticipating infrastructure costs based on market trends.
• Risk Management: Understanding vendor stability and market dynamics.

The next phase of AI adoption will focus on production deployment, operational excellence, and cost optimization—areas where DevOps expertise becomes critical. Companies that can operationalize AI at scale will have a significant competitive advantage, and the stock market is already reflecting this reality.

Bottom line: The AI revolution isn’t just about technology—it’s reshaping entire markets. DevOps teams that understand these dynamics can make better infrastructure decisions and position their organizations for success in the AI era.

TL;DR

• Micro made AI threat modeling real with a formal Secure Future Initiative (SFI) update, an AI Red Team playbook, and default guardrails in Azure AI Studio.
• Their stack is layered: harden data sources, instrument training/inference pipelines, and close the loop with SOC-grade telemetry (Security Copilot).
• You can replicate 80% of it today using reproducible builds, signed checkpoints, isolation boundaries for inference, and policy-as-code guardrails around prompts and outputs.

What Micro already shipped

1. Secure Future Initiative for AI (2023 → 2024 refresh)
   Micro took the SFI controls—passwordless identity, secret scanning, memory-safe rewrites—and applied them to model hosting. Every Azure OpenAI endpoint now inherits Conditional Access policies, managed identity, and default customer-managed keys.

2. AI Red Team + Responsible AI Standard v2
   Their in-house Red Team published threat trees for prompt injection, data exfiltration, and model theft. Those trees plug straight into Microsoft’s Responsible AI Standard (transparency notes, safety evaluations, abuse monitoring) and gate releases of Copilot, Bing, and Phi-3.

3. Security Copilot + Defender connectors (GA October 2024)
   The SOC tool now ingests GPT-4-based summaries plus native signals from Defender for Cloud, Purview, and Sentinel. For AI workloads, you can trace a risky prompt to the Azure resource, user identity, and token permissions in one pane.

4. Prompt Shields + Azure AI Content Safety
   Build 2024 introduced Prompt Shields (jailbreak/indirect prompt filtering) composed with Content Safety classifiers. Micro exposed both as policies in Azure AI Studio so platform teams can enforce “no PII egress” or “no code execution instructions” without rewriting their apps.

5. Confidential inference + watermarking
   The ND H100 v5 VMs ship with Intel TDX/AMD SEV-SNP, so models and prompts stay encrypted in use. At the same time, Micro partnered with C2PA to watermark Copilot outputs, making provenance verification part of the default toolchain.

Build the same layers in your stack

Recommended controls (copy/paste friendly)

apiVersion: policy.sigstore.dev/v1beta1
kind: AttestationPolicy
metadata:
  name: ai-model-release
spec:
  subjects:
    - pattern: ghcr.io/acme/llm-serving:*
  attestations:
    - name: model-integrity
      predicateType: https://slsa.dev/provenance/v1
      policy:
        script: |
          rule integrity_is_signed:
            condition:
              input.provenance.builder.id in ["https://micro.ai/redteam/builder", "https://acme.ai/builder"]

Wire this into your CI so only images with trusted SLSA provenance—and, optionally, Micro’s shared builders—can reach production.

# Sandbox tool execution the same way Micro isolates Copilot plug-ins
kubectl apply -f - <<'EOF'
apiVersion: gvisor.dev/v1
kind: SandboxedRuntime
metadata:
  name: llm-tools
spec:
  runtimeHandler: runsc
  annotations:
    workload: "ai-tools"
EOF

Checklist before you ship the next model

1. Map assets – training data, fine-tune weights, prompt templates, eval harness.
2. Decide trust boundaries – separate data prep, training, inference, and retrieval.
3. Instrument every hop – capture prompts, tool calls, and policy verdicts as structured events.
4. Rehearse failure – run Micro-style Red Team drills for jailbreaks, data poisoning, and lateral movement.
5. Close the loop – feed detections back into guardrails and developer training.

Treat Micro’s work not as a vendor story but as a template. If their Copilot stack can prove provenance, enforce policy, and keep SOC eyes on every inference, so can yours—just swap in your clouds, your secrets engine, and your favorite LLM runtime.

Quick snapshot

• Open training stack - Pure PyTorch with Composer callbacks (layer freezing, gradient clipping, mixup-style augmentation) wired for DeepSpeed ZeRO-3 and PyTorch FSDP.
• Model family - MPT-7B/30B, DBRX Instruct, and assorted long-context checkpoints (32K) all advertise “Trained with Composer,” so weight provenance is clear.
• License - Apache 2.0 across the board, which means redistribution, fine-tuning, and even hosted SaaS offerings are allowed with attribution.
• Optimizations - FlashAttention 2, fused RMSNorm, rotary embeddings + ALiBi, activation checkpointing, and muParam scaling are first-class features in the training recipes.

Why it matters for DevOps and platform teams

1. Transparent supply chain - Composer ships the full registry of algorithms (progressive resizing, stochastic depth, etc.) in plain config files, so you can document exactly how the base model was trained.
2. Predictable scaling - Because the stack targets commodity PyTorch + NCCL, every knob you already tune for other GPU workloads (batch size, tensor parallelism, ZeRO stage) applies here as-is.
3. Native tool hooks - The repos include reference Helm charts, vLLM configs, and Terraform snippets for Databricks Model Serving, which means GitOps teams can slot Composer weights into their existing release trains.
4. Fine-grained guardrails - You keep the tokenizer, system prompts, reward models, and safety adapters, so policy tuning lives with your infra instead of an upstream vendor.

Architecture cheatsheet

Tokenizer   : SentencePiece 32k vocab (shared by MPT + DBRX so fine-tunes stay compatible)
Backbone    : Decoder-only transformer, SwiGLU feed-forward, RMSNorm normalisation
Attention   : Multi-head attention with ALiBi for long context (up to 32,768 tokens in DBRX)
Training    : bfloat16 mixed precision, cosine LR decay, muParam scaling, EMA checkpoints
Inference   : vLLM / Text-Generation-Inference templates + KV cache partitioning for >30 concurrent streams

Because the stack is openly documented, you can reproduce the base checkpoint or fork the training recipe (data mixtures, optimizer, evaluation harness) whenever compliance needs proof.

Getting it running (Kubernetes flavored)

1. Grab the weights

   huggingface-cli download mosaicml/mpt-7b-instruct --local-dir ./models/mpt-7b-instruct
   

2. Package an image - Base off nvcr.io/nvidia/pytorch:24.04-py3, add vllm==0.4.x, copy your tokenizer + weights, and expose a /generate endpoint.
3. Deploy with autoscaling - Use KServe or OpenShift Serverless, wire GPU node selectors, and let KEDA watch Kafka or Redis queues for traffic bursts.
4. Observability - Attach OpenTelemetry spans around decode loops; Composer’s throughput varies a lot with top-k/top-p choices, so exporting tokens/sec and waiting-room time keeps SREs sane.

Fine-tuning playbook

Because Composer exposes full training traces and data-mixture manifests, you can reproduce the base checkpoint if auditors ever demand it or swap out sensitive corpora before re-training.

Risk checklist before shipping

• Load-test 99th percentile latency with production prompt lengths; Composer is fast, but KV cache thrash happens when users paste entire runbooks.
• Run red-team prompts against your tuned model; the Apache 2.0 license lets you modify safety adapters, which means you must own the residual risk.
• Version both the weights and tokenizer in artifact storage; mismatched vocab files are the most common source of “gibberish output” incidents.
• Bake in drift detection (embedding cosine distance or perplexity over a canary dataset) so you notice when fine-tuning nudges the model off the rails.

Bottom line: Composer LLM isn’t a single secret model–it’s a transparent recipe plus a family of open checkpoints that play nicely with modern DevOps pipelines. With a little YAML and observability discipline, you can run it alongside existing service meshes and treat generative AI like any other production workload.