AgentCost
AI Cost Observability Infrastructure
Teams adopt LLM agents and lose the thread between an invoice and the code that produced it. AgentCost instruments the client libraries an application already uses, captures token and cost data at the call site, and attributes every event to the agent, workflow, step and tool it came from. Instrumentation is two lines of Python; the rest happens transparently on a non-blocking path.
- Python
- FastAPI
- PostgreSQL
- SDK Design
- LLM Observability
- Next.js
The problem
LLM spend arrives as a single opaque provider invoice. When a bill jumps, there is nothing connecting the number to the agent, workflow or model choice responsible for it. Existing APM tooling traces latency and errors well, but has no concept of tokens, model pricing, or the nested agent runs that modern LLM applications are built from. The practical questions have no answer without custom instrumentation in every codebase: which agent is expensive, which step is wasteful, whether a classification task is quietly running on a frontier model.
The approach
Instrument at the client library rather than the application. AgentCost patches the provider SDKs already imported by the process, so existing call sites are untouched and no refactor is required to adopt it. Each intercepted call is priced against a model catalogue and tagged with the surrounding agent, workflow and step, tracked through contextvars so attribution survives concurrency. Events accumulate in memory and flush in batches on a background path, keeping the request hot path free of network work. The same event stream can stay local for development and CI, or ship to the hosted backend for dashboards, budgets and anomaly detection.
Architecture
A pass-through instrumentation path: intercept, price, attribute, batch, ingest, query. The application never blocks on AgentCost.
- Agent / service codeSource
- Provider client librariesSourceOpenAI, Anthropic, Gemini, LangChain
- Monkey-patched methodsProcess
- Token + cost resolutionProcesstiktoken, model price catalogue
- Streaming accumulatorProcess
- contextvars scopeProcessagent / workflow / step / tool
- Metadata + outcome tagsProcess
- In-memory event bufferStore
- Batched async dispatchProcesssize or interval trigger
- Graceful shutdown flushProcess
- FastAPI ingest endpointProcess
- Project key authProcess
- PostgreSQL event storeStore
- Local mode (in-process)Storedev and CI
Any Python process making LLM calls: a service, an agent loop, a LangChain pipeline, a batch job. Call sites are unmodified.
Technical decisions
Patch the client, don't wrap it
A wrapper API means every call site in every service has to change, which makes adoption a migration project and guarantees partial coverage. Patching the provider client at init instruments code the team has already written, including calls made deep inside frameworks like LangChain that the application never touches directly. The tradeoff is a tighter coupling to provider SDK internals, which has to be tracked as those libraries change.
contextvars over thread-locals or explicit passing
Agent code is overwhelmingly async, and a single event loop interleaves many runs. Thread-locals get this wrong immediately, and threading an explicit context object through every function defeats the point of transparent instrumentation. contextvars is the primitive that actually matches the execution model: each async task inherits and can override its own scope.
Batch asynchronously, never on the hot path
An observability tool that adds latency to inference calls gets removed. Events are appended to an in-memory buffer and flushed by a background worker on a size or interval trigger, so the calling code pays roughly an append. The cost of that choice is durability, which is why shutdown explicitly drains the queue rather than relying on process exit.
Local mode as a first-class path
Requiring a backend to see anything makes the tool impossible to evaluate and useless in CI. Running the same collection path in-process, with no network, means the SDK can be tried in a scratch script and asserted against in a test suite. That is also what makes pre-deployment cost analysis possible at all.
System capabilities
8 of 8 implemented. The rest are labelled with what they actually are.
Zero-refactor instrumentation
ImplementedProvider clients are patched at init. Existing call sites keep working untouched; adoption is an import and one call.
Hierarchical attribution
ImplementedCost resolves per run, step and tool rather than per API key, so an expensive agent can be isolated from a cheap one.
Concurrency-safe context
ImplementedAttribution rides on contextvars, keeping tags correct across async tasks and threads in the same process.
Streaming support
ImplementedStreamed completions are accumulated so token usage and cost are captured with the same accuracy as unary calls.
Non-blocking transport
ImplementedEvents buffer in memory and dispatch in background batches, with a shutdown flush so short-lived processes don't drop data.
Local and cloud modes
ImplementedThe same SDK can hold events in-process for development and CI, or ship them to the hosted backend.
Budgets and anomaly detection
ImplementedMonthly budgets with threshold alerts, plus detection of spend that departs from the established pattern.
Pre-deployment cost analysis
ImplementedLocal-mode events let a CI run measure what a change costs before it ships.
Technology stack
SDK
- Python
- tiktoken
- httpx
- contextvars
- Monkey-patching
Backend
- FastAPI
- async SQLAlchemy
- PostgreSQL
- Docker
Dashboard
- Next.js
- React
- Tailwind CSS
- Recharts
Challenges
The parts that were genuinely hard. Pretending everything went smoothly makes the rest less believable.
Usage data arrives last in streaming responses
For streamed completions, token counts are only known once the stream completes, and the stream may be abandoned partway. The interceptor has to wrap the iterator itself, accumulate as chunks arrive, and still emit a correct event when the consumer breaks out early.
Attribution under concurrency
When many agent runs share one event loop, naive global state attributes cost to whichever run happened to set it last. Getting this right meant modelling scope as a stack carried per-task rather than per-process.
Model pricing drifts constantly
Providers add models, rename them and change prices. Cost calculation is only as good as the catalogue behind it, which makes pricing data a maintained artefact rather than a constant, and makes unknown-model handling a real design question rather than an edge case.
Patching without breaking the host application
Instrumentation sits in the middle of somebody else's request path. It has to preserve signatures, return types and exception behaviour exactly, and fail open. An error inside the SDK must never surface as an error in the application's LLM call.
Current status
Built and running. The Python SDK, ingestion backend and dashboard are implemented, the hosted service is live and free to use, and the project is open source under the MIT licence.
Next steps
- Broaden framework coverage as agent libraries change shape
- Deeper optimisation suggestions — flagging work running on more model than it needs
Have a difficult problem worth building?
We’re interested in ambitious software, AI systems, research collaborations and real problems where careful engineering actually changes the outcome.