Cutting AI Cost per Document by 60% at the Same Quality for a UK Energy Supplier
A controlled side by side test for a UK energy supplier, which found a model that matched the quality of the incumbent at around 60% lower inference cost per document, with spend capped by design from the first request.
Company Overview
The client supplies power and gas to businesses across the UK. Much of its sales and onboarding work starts with documents sent in by customers and brokers, which have to be read, checked and keyed in before anything can move.
AI can take much of that reading off the team, but only if the business can see what each answer costs, trust its quality, and change the model later without starting again.
Problem Statement
Model prices are hard to compare on a price list. Models count tokens differently, output and reasoning can be a large part of the bill, and a cheaper model that needs retries or extra checking can end up costing more. The supplier needed to manage the decision, not just the model price: who owns it, what each document costs, where the limits sit, and what proves the value.
Outcome Delivered
Both models recovered 8 of 9 reference values, and the challenger did it for about 60% less per document: £0.0036 against £0.0090. The supplier now has a repeatable way to choose models on cost per useful outcome, and a person still reviews the cases where both models were over-confident.
Executive Overview
A UK energy supplier wanted to use AI to pull key fields, such as company, address and named person, out of business documents, with evidence for every answer. The question was not which model is best in general, but which model does this task at acceptable quality and cost.
We ran a controlled side by side test of two models, GPT-5 mini as the baseline and DeepSeek V4 Flash as the challenger, both hosted in the EU on Azure. Both recovered 8 of 9 reference values, and the challenger cost about 60% less per document. Spend was capped throughout, so the test itself could never run away with the budget.
Our Approach
We put rules first and used AI only where it adds value. Each document is read once, with its page references kept. Rules then validate formats, find candidate values and reject impossible ones. Only then does the model resolve meaning and uncertainty, and every answer is checked against its source with a person kept at the point of decision.
The test ran in six steps. We set the boundary, EU-hosted with a £50 ceiling and budget reserved before each request. We created reference answers kept apart from the prompt, using 16 documents split into 7 for development and 9 held back. We froze the job so both models saw the same text and instructions, ran a small qualification test before spending on real documents, limited each attempt to one call with an 8,192 token cap and no retries, and measured the whole result: tokens, time, cost, accuracy, evidence and how each model handled uncertainty.
Challenges
Challenges Faced
Every engagement starts with a clear-eyed view of what was actually getting in the way, because you cannot fix what you cannot see. Here we set out the real constraints the client was working within, the costs and risks they were carrying, and the gaps that needed closing before any meaningful progress could be made.
Let's Transform Your Enterprise Today
Send an EnquiryPrices that do not compare
Each model counts tokens differently and bills reasoning separately, so the list price says little about the real cost of a document.
Quality has to be proven
A cheaper model is only cheaper if it gets the answers right, so quality had to be measured against reference answers rather than assumed.
Test spend had to stay capped
The evaluation itself needed a hard limit, so a failed run or a loop could never be quietly charged again and again.
Over-confident answers
Both models were over-confident on a deliberately contradictory broker name, which showed that a person still needs to review the hard cases.
Deployed solutions
What We Did
A look at how we approached the work, from the first discovery sessions through to the changes we put in place, and the thinking behind each decision along the way. The aim is always the same, to deliver something measurable that the client owns and can build on, rather than a one-off fix that fades once we leave.
Hard token limits
Every response was capped at 8,192 output tokens, with actual usage recorded underneath, so routing, caching and output caps can be tested on real volumes.
No automatic retries
A failed quality check is saved for analysis rather than charged again, and runs stop on repeated errors.
Stage gates
Extraction is saved before the model runs, and model output is saved before the rules run, so paid work is never repeated and nothing reaches production without passing a gate.
Structured answers
Named fields, a level of uncertainty and source evidence for every value, checked automatically before a person relies on them.
One baseline, one challenger
A baseline run with its failures studied, then the most useful lower cost alternative tested on the same job, so new models are evaluated before anyone switches.
Cost per useful outcome
The decision rests on cost per accepted answer, not cost per million tokens, and the same test pack can be re-run whenever prices or models change.