In-band and out-of-band — what your AI observability can know depends on where it stands

Every AI observability tool stands somewhere, and where it stands determines what it can know. There are exactly two places to stand.

Out-of-band tools instrument your code: an SDK wrapper, a decorator, a callback handler, a tracing library. They see what your application believes is happening — which chain invoked which model, with what intent, in which trace — and they report it after the fact, alongside the work.

In-band tools stand in the request path itself: a proxy the traffic traverses. They see what actually crossed the wire — the literal bytes of the request, the provider's literal response, the real latency between them — and they see it while it is happening, which means they can also do something about it.

Vigil is in-band, and this post is the honest version of what that buys and what it costs — honest because the costs are real, and because most comparisons in this market are two positions talking past each other as if they were substitutes. They are not. They are different vantage points on the same traffic, and they know different things.

What only the wire can tell you#

What was sent, versus what your code meant to send. Frameworks assemble prompts: they inject headers into system prompts, reorder tool arrays, append context. Out-of-band instrumentation records the inputs to that assembly; the wire carries its output — and the gap between them is precisely where cache failures live. A timestamp your framework helpfully added or a tool array that reorders between replicas is invisible at the SDK layer, where the code looks identical on every call, and glaring at the byte layer, where the prefix hash never repeats.

What the provider answered, versus what your SDK surfaced. A streamed call that reports zero tokens because a flag was absent, an error body that arrived as an edge's HTML rather than provider JSON, a 524 from an intermediary the SDK renders as a generic failure — out-of-band tooling inherits the SDK's view. In-band, the wire is the record: token counts from the actual usage chunks, failures classified from the actual body.

Everything, or only what remembered to instrument itself. A wrapper covers the code that imports it. The cron job someone wrote with a raw fetch, the second service, the contractor's script with your API key — none of them announce themselves to an SDK. A proxy in the path sees whatever traverses it, whoever wrote the caller — coverage is a property of routing, not of discipline. The corresponding honesty: a proxy sees only what is routed through it, and a caller that bypasses the URL is invisible. Neither position sees everything; the difference is that routing is auditable in one place and instrumentation discipline is auditable everywhere.

And only in-band can act. Observation aside, a proxy can change outcomes on the way through: place a cache breakpoint where the auth scheme permits it, inject the usage flag a streamed call needs, convert an edge death into a parseable error. An out-of-band tool can recommend; the request has already left.

What out-of-band knows that we structurally cannot#

The wire does not carry your intent. A proxy sees forty calls costing eleven dollars from one agent fingerprint; it cannot see that they were one LangGraph run, which node looped, or which retrieval step fed which generation. Chain topology, span hierarchy, the why behind the traffic — that is application knowledge, and only instrumentation standing inside the application can have it. If your daily question is "which node in my graph is misbehaving?", an out-of-band tracer answers it and Vigil does not. If the question is "what is this costing, why is the cache missing, and what actually failed?", the wire is the only witness whose testimony is first-hand. Plenty of teams reasonably run both — the positions compose, because they never occupy the same ground.

The cost of standing in the path, stated plainly#

In-band means the traffic depends on you, and that dependency is the tax on everything the position knows. It is why the proxy's failure posture is absolute — a plan limit, a detection, a logging failure may degrade what Vigil does, never what your request does — and why the fail-open-versus-fail-closed decisions get argued direction by direction, and why telemetry is forbidden from stalling a stream it is reading. An out-of-band tool's worst day loses you some spans. An in-band tool's worst day is an outage — what happens when Vigil goes down is a question we answer in writing, because standing in the path means owing that answer before anyone asks.

Where you stand is what you know, and the tools are named by their vantage point more than by their feature lists. The wire cannot tell you what your code meant, and your code cannot tell you what the wire carried.