Spectra
Document Intelligence and Retrieval
Spectra ingests documents, chunks and embeds them, and answers questions against them with citations back to source. The distinguishing constraint is that it runs its language model locally rather than calling a hosted API, which changes the engineering problem: latency, model size and prompt budget all become design constraints instead of billing lines.
- FastAPI
- ChromaDB
- Hugging Face
- Ollama
- React
- RAG
The problem
A general-purpose assistant cannot answer questions about a document it has never seen, and pasting long documents into a context window is both lossy and expensive. The useful version of this problem is narrower than 'chat with your files': answers have to be grounded in specific retrieved passages, scoped to the document the user actually means, and returned fast enough to feel conversational. Retrieval quality is doing the real work. A weak chunking or embedding decision cannot be recovered by a better prompt.
The approach
Two separate paths that meet at the prompt. On ingestion, documents are parsed, split into overlapping chunks, embedded with a Hugging Face sentence-embedding model and written to ChromaDB alongside metadata identifying their source document. On query, the question is embedded with the same model, matched against the vector store with a metadata filter that constrains results to the selected document, and the retrieved passages are assembled into a bounded context for a locally served model through Ollama. Generation is streamed token by token so the interface responds immediately rather than after a full completion.
Architecture
An ingestion pipeline and a query pipeline sharing one embedding model and one vector store. Symmetry between the two is what makes retrieval work.
- Document uploadSource
- Document registryStoreid, title, timestamps
- Text extractionProcess
- Overlapping chunkerProcess
- Hugging Face embedding modelModel
- ChromaDB collectionStore
- Chunk metadataStoredocument id, position
Documents enter through the interface and are registered with an identifier that every downstream chunk carries.
Technical decisions
ChromaDB over a hosted vector service
The system was built to run entirely on a local machine, which rules out a managed vector database on principle rather than on price. ChromaDB embeds directly in the Python process, persists to disk, and supports the metadata filtering that per-document scoping depends on. At this scale the operational simplicity is worth more than the throughput a dedicated service would add.
Local inference through Ollama
Running a local model keeps document content on the machine that owns it and removes per-query cost from the design entirely. What it adds is constraint: a small model has a smaller effective context and less tolerance for a sloppy prompt, which forced retrieval and context assembly to be good rather than merely adequate. Answer quality on hard questions is the price paid for that.
Metadata filtering rather than one store per document
Isolating documents by giving each its own collection would guarantee scoping, but makes cross-document search impossible later and multiplies the state to manage. Tagging chunks with a document id and filtering at query time keeps a single index while still allowing strict scoping, and leaves the door open to querying across a library.
Stream the response
Local generation is slower than a hosted frontier API. Waiting for a full completion before rendering anything makes that gap feel much worse than it is. Streaming turns time-to-first-token into the number the user actually perceives.
System capabilities
6 of 7 implemented. The rest are labelled with what they actually are.
Document upload and ingestion
ImplementedDocuments are uploaded, parsed and processed into a queryable index.
Chunking and embedding pipeline
ImplementedOverlapping chunking with Hugging Face sentence embeddings, applied identically at index and query time.
Vector storage with metadata
ImplementedChromaDB persistence with per-chunk document metadata backing scoped retrieval.
Per-document querying
ImplementedRetrieval can be constrained to one document so answers cannot draw from unrelated sources.
Local model inference
ImplementedGeneration runs against a locally served model through Ollama rather than a hosted API.
Streaming chat interface
ImplementedAnswers stream into the interface as they are generated.
Retrieval evaluation
In progressSystematic measurement of chunking and embedding choices against a labelled question set.
Technology stack
Retrieval
- ChromaDB
- Hugging Face embeddings
- Chunking pipeline
Inference
- Ollama
- Gemma
- Streaming generation
Backend
- Python
- FastAPI
Interface
- React
- Vite
- Tailwind CSS
Challenges
The parts that were genuinely hard. Pretending everything went smoothly makes the rest less believable.
Chunk boundaries decide answer quality
Most bad answers traced back to retrieval rather than generation: a chunk that split a definition from its subject, or one large enough that its embedding averaged into vagueness. Tuning size and overlap was the highest-leverage work in the project.
Context budgets are real with local models
A locally served model leaves far less room than a hosted frontier one. Deciding how many passages to include, and what to drop when they don't fit, is an explicit ranking problem rather than something the context window absorbs.
Streaming through the whole stack
Token-by-token delivery has to survive every layer — model runtime, FastAPI response, and the React client — without buffering somewhere in the middle and collapsing back into a single delayed chunk.
Current status
A working prototype. The full ingestion and retrieval path is implemented end to end and runs locally, including streamed generation. Retrieval quality evaluation is in progress; the system has not been hardened for multi-user or production deployment.
Next steps
- Measure retrieval quality against a labelled question set instead of by inspection
- Cross-document querying over a whole library, not one document at a time
- Citation surfacing that links each claim back to its source passage
Have a difficult problem worth building?
We’re interested in ambitious software, AI systems, research collaborations and real problems where careful engineering actually changes the outcome.