Before Buying an AI Solution: How to Compare the Real Cost of the Pilot and Live Operation

Why the pilot cost misleads

A pilot usually runs on a limited volume of requests, with data the team has prepared in advance and intensive support from a vendor that wants to win the contract. If the purchasing decision rests on the pilot cost, the gap appears a few months into live operation. Ask the vendor to separate pilot prices from operating prices in the proposal, because some offers price the pilot at a discount or free of charge.

The UK government’s AI Playbook, published in 2025, points out that these systems can carry considerable compute costs and that maintaining them requires ongoing investment. ^[2] NIST’s Generative AI Profile notes that running these systems to generate output is resource-intensive, and that they usually depend on many third-party components. ^[3] In other words, part of the cost varies with volume, and part comes from outside the proposal in front of you.

What goes into the total cost of ownership?

Add up every item over the same period, ideally the full contract term rather than the first year alone.

The UK AI Playbook recommends giving suppliers guidance on budget that takes hidden costs into account, and stating in the requirements any integration with associated technologies or services. ^[2] Integration, data preparation and permissions are real costs even when they do not appear on the price quote.

Do not forget the internal costs nobody sends an invoice for: the service owner’s time managing the vendor, review by the cybersecurity and data protection teams before connection, user training and updating written procedures. Estimate them in expected working hours, even roughly, because ignoring them makes every solution look cheaper than it is. If you add a contingency for what you do not yet know, show it as a visible line with its percentage and rationale documented, not as an amount spread quietly across the other items.

If pricing depends on the volume of text or pages processed, ask the vendor to estimate the consumption of a typical document from your own documents, then test a long document and one with attachments, because consumption can multiply with document length.

Use a common unit: the cost per accepted output

The advertised unit fee is not enough for comparison. The solution with the lowest fee can turn out the most expensive if its output needs more review. So define the accepted output first, such as a completed invoice that has passed review, then calculate the cost of reaching it:

Cost per accepted output = (usage fees + review and correction time × cost of time + allocated fixed costs) ÷ number of accepted outputs

Measure the referral rate and the average review time in the pilot itself, using the real data that will arrive in live operation, not the vendor’s samples.

Hypothetical example

A hypothetical organisation is comparing two invoice extraction solutions. Solution A charges 0.20 currency units per invoice but refers 30% of invoices for review because it reads Arabic layouts poorly. Solution B charges 0.35 and refers 10%. Review takes 4 minutes on average at a cost of one currency unit per minute.

In a normal month of 10,000 invoices, Solution A costs 2,000 in fees and 12,000 in review, a total of 14,000. Solution B costs 3,500 in fees and 4,000 in review, a total of 7,500. The gap widens in the peak month at period close. The cheaper solution on the quote costs nearly twice as much in operation. Fixed costs are left out of the example because they are assumed to be equal.

Then test how sensitive the result is to its weakest assumption, usually the referral rate. If Solution B’s referral rate rose to 20% in live operation instead of 10%, its monthly cost would be 3,500 in fees and 8,000 in review, a total of 11,500, and it would still be cheaper than Solution A. If the ranking holds when the weakest assumption changes, the decision is sounder.

Test the effect of changing volume

Calculate the cost in at least three scenarios: normal volume, expected peak and growth over the contract term. Then check what happens when volume exceeds what is included: do fees rise in tiers, does performance slow down, does the service stop?

For complex outsourcing projects in central government, the UK Sourcing Playbook requires a Should Cost Model to be produced early, comparing whole-life cost across in-house delivery, buying from the market and a mixed model, and protecting against bias towards low-cost bids. ^[1] The same idea is useful for AI: estimate what the solution should cost before you read the proposals.

When do you move from pilot to live operation?

Set the success criteria before the pilot begins, not afterwards. The Sourcing Playbook describes a pilot as typically the final test before full rollout, and calls for testing proportionate to the project’s size, complexity and uncertainty, with clear success criteria. ^[1]

Practical criteria include a referral rate within a set limit, acceptable processing time at peak, and no critical errors in financial fields or permissions. The UK AI Playbook recommends fully testing the product before deployment, keeping regular checks on the live tool, and monitoring for drift and bias. ^[2] Those checks are an ongoing cost that belongs in the estimate.

At the end of the pilot, hold a formal decision point attended by the service owner, procurement and cybersecurity, where actual results are compared with the written success criteria and one of three decisions is taken: move to live operation, extend the pilot under defined conditions for a set period, or stop. Stopping is a legitimate outcome of a pilot, not a failure of it.

How to use the estimate in the contract

An estimate that does not make it into the contract does not protect you. Record in it:

• Assumptions: the volume the price covers, the expected referral rate and the required quality level.
• Price limits: a cap on increases at renewal, and overage charges. The Sourcing Playbook advises avoiding fixed-price mechanisms where scope is not fixed, and including indexation where suppliers cannot control costs such as inflation. ^[1]
• Service levels and monitoring: NIST highlights vendor contracts and service level agreements, and continuous monitoring of third-party systems once deployed. ^[3]
• Updates: who bears the cost of retesting when the vendor changes the model.
• Data: whether the vendor may use your data or outputs to improve its model, and on what terms.
• Exit: the cost of extracting and transferring your data at the end of the contract, intellectual property rights in anything developed during it, and a fallback that keeps the service running if the solution stops. ^[2]

The Sourcing Playbook stresses that bid evaluation should consider whole-life cost rather than the lowest price, ^[1] which is the essence of the whole comparison.

Applying this in the Gulf

Calculate the effect of taxes, currencies and transfer by country and contract terms, since VAT rates and rules differ between Gulf states. Check the data hosting and classification requirements in your country, as they may require a hosting option that costs more than the base proposal. If the solution is hosted outside the country, estimate the cost of moving to local hosting should requirements later demand it. In government bodies, contracting is subject to each country’s government procurement rules, and the pilot estimate should not be treated as a financial commitment.

Conclusion

Compare AI solutions on the cost per accepted output over the full contract term, at normal and peak volumes. Include review, integration, monitoring and exit in the calculation, and fix the assumptions in the contract before signing.