Order Intelligence
Operational Intelligence for Fragmented Workflows
In many operations businesses, the real state of an order lives across email threads, chat messages and phone notes, and only exists coherently in somebody's head. This is a design for an extraction and coordination layer that reads those messages, proposes structured updates, and routes anything uncertain to a person before it changes a record. It is specified and architected; it has not been built.
- LLM Extraction
- Human-in-the-Loop
- Workflow Systems
- System Design
The problem
Operational work spreads across channels that were never designed to hold state. An order is amended in a reply, confirmed by phone, and adjusted again in a chat message — and the authoritative version is whatever the person handling it remembers. The cost is invisible until it is not: duplicated effort, missed amendments, and no way to answer basic questions about status without asking someone. Conventional automation fails here because the input is unstructured natural language, and naive LLM automation fails differently — a model that silently writes a wrong quantity into a record is worse than no automation, because the error is now trusted.
The approach
Separate extraction from commitment. Incoming messages are normalised into a common representation, an LLM extracts candidate structured updates against an explicit schema, and each candidate carries a confidence signal. High-confidence updates that fit the expected shape can apply directly; anything ambiguous, contradictory or unusual goes to a review queue where a person confirms or corrects it. Order state is held as an append-only event log, so every change retains its provenance — which message caused it, whether a model or a person committed it — and a wrong extraction is a correctable entry rather than a silently overwritten field.
Architecture
Proposed design. The load-bearing idea is that the model proposes and a person disposes — nothing uncertain reaches committed state unreviewed.
- Email threadsSource
- Chat messagesSource
- Call notesSource
- Unified message recordProcess
- Thread + entity linkingProcess
- Schema-constrained extractionModel
- Confidence scoringModel
- Confidence routingProcess
- Human review queueInterfaceconfirm or correct
- Append-only event logStore
- Derived order stateStore
- Provenance trailStore
- Operational dashboardOutput
- Exception surfacingOutput
Operational communication arrives across several unstructured channels, each with its own format and conventions.
Technical decisions
The model proposes; a person commits
Extraction from real operational language is not reliable enough to write directly into a system of record, and a confidently wrong value is more damaging than a missing one because downstream work will trust it. Making every extraction a proposal, and gating uncertain ones behind review, means the failure mode is a queue item rather than a corrupted order.
Event log rather than mutable records
When a model participates in updating state, being able to answer 'why does this field say that' is not optional. An append-only log keeps the causal chain — which message, which extraction, which person confirmed it — and makes a bad extraction a correctable entry rather than an overwrite with no history.
Constrain extraction to a schema
Free-text output has to be parsed, and parsing failures become silent data loss. Constraining generation to an explicit schema turns a large class of model errors into validation errors that can be caught and routed rather than absorbed.
System capabilities
0 of 5 implemented. The rest are labelled with what they actually are.
Multi-channel message normalisation
Designed, not builtDesigned approach for reducing email, chat and notes to one message representation.
Schema-constrained extraction
Designed, not builtDesigned extraction producing candidate updates against an explicit schema rather than free text.
Confidence-based routing
Designed, not builtDesigned policy for which updates apply automatically and which require human confirmation.
Human-in-the-loop review
Designed, not builtDesigned review queue where a person confirms or corrects proposed updates before they commit.
Event-sourced state with provenance
Designed, not builtDesigned append-only model retaining which message and which actor produced every change.
Technology stack
Proposed extraction
- LLM structured extraction
- Schema-constrained output
- Confidence scoring
Proposed state
- Event-sourced order state
- Audit trail
- Reconciliation
Proposed interface
- Review queue
- Exception surfacing
- Operational dashboard
Challenges
The parts that were genuinely hard. Pretending everything went smoothly makes the rest less believable.
Confidence is hard to calibrate
The routing policy is only as good as the confidence signal behind it. A model's self-reported certainty is not reliably calibrated, so deciding what may bypass review is an open problem rather than a configuration value.
Review load determines whether it helps
Route too much to humans and the system adds work instead of removing it; route too little and errors reach committed state. That threshold is the design's central risk.
Real messages are messy and contradictory
Operational language is elliptical, references shared context, and frequently contradicts an earlier message in the same thread. Resolving which statement is current is often genuinely ambiguous, not merely hard to parse.
Current status
A concept. The problem, architecture and human-in-the-loop model are specified; no implementation exists. It is published here as a piece of systems thinking, not as a product — there are no deployments, users or customers.
Have a difficult problem worth building?
We’re interested in ambitious software, AI systems, research collaborations and real problems where careful engineering actually changes the outcome.