Why We Stopped Renting AI and Built Our Own

Inside EVA the platform we built to own our LLM stack end to end

For the better part of a year, a core piece of our product ran on someone else's model. We'd send a structured payload to a general-purpose LLM API, get generated content back, and hope it was accurate, correctly formatted, and appropriately personalized. At small scale, that's a workable architecture. At real production scale, it stopped being one.

Four problems compounded on each other. System prompts couldn't reliably encode the nuance of our own business taxonomy, so outputs defaulted to generic, repetitive language. Structural reliability was inconsistent enough that a meaningful share of responses needed manual exception handling before they could enter an automated pipeline. Every request sent detailed data outside our organizational boundary, which sat uneasily against our own data governance standards. And the operational ceiling the time and cost of processing a large daily batch sequentially through an external API had no internal path to improvement. We could only ask the provider for more, or pay more per request.

The Fork In The Road

The obvious fix was a single fine-tuning project:

pick a base model, adapt it to our domain, ship it, and repeat the exercise whenever the model aged out or a new use case appeared. That would have solved this problem. It wouldn't have solved the next one.

So we asked a different question:

what if we treated this as a platform problem rather than a one-time model problem?

What if data pipelines, training, evaluation, serving, and continuous improvement were all built once, as reusable infrastructure, so that any future use case could connect to it rather than needing its own from scratch?

That decision became EVA.

What we built

EVA is a five-phase, containerized pipeline data engineering, supervised fine-tuning, evaluation and quality governance, inference and serving, and continuous alignment with each phase independently deployable and communicating through shared artifact storage rather than tight coupling. That decoupling matters more than it sounds: it means we can retrain against new data without touching the serving layer, and roll a new adapter into production without redeploying anything.

The platform is model-agnostic by design. The base language model is a configuration choice, not an architectural one today it's a mid-size open-source instruction-tuned model, chosen specifically because it held up reliably at our required context length on the hardware we run, without the memory pressure a larger model introduced. Tomorrow it could be a different model entirely, with no rework to the pipeline around it.

We also broke from standard NLP evaluation practice. Metrics like ROUGE or BERTScore measure similarity to a reference text useful for some tasks, irrelevant for ours. What matters for us is whether an output is structurally valid, factually grounded in the real input data, and correctly classified, regardless of whether it resembles any training example. So we built our own evaluation suite around the dimensions that actually define quality for our use case: schema compliance, classification accuracy, personalization, hallucination, evidence grounding, and length compliance. On serving, we run a high-throughput inference engine with hot-swappable model adapters and a tiered fallback chain, so a single point of failure never takes the system down it degrades gracefully instead.

The numbers

Post-fine-tuning, against the baseline we started from:

• Schema compliance: a sizable minority-failure rate → 100%
• Hallucination: significantly reduced.
• Daily generation time: hours → minutes (10x improvement)
• Per-item latency: seconds → a fraction of a second (10x improvement)
• External data exposure: eliminated from the primary path entirely

What's next

Production deployment wasn't the finish line. EVA includes a built-in human feedback loop reviewers rate real production outputs, and that feedback becomes the training signal for a preference-alignment stage (DPO) layered on top of the fine-tuned model. No synthetic labels. Every deployment generates the signal to improve the one after it..