Why Data Engineering Is the Real AI Bottleneck

In UK mid-market CPG and logistics, AI programmes stall at the data engineering layer, not the model. Fragmented sources, fragile pipelines and unclear ownership of data quality are what keep AI out of production. Measuring data readiness first, before model selection, is the single highest-return step a leadership team can take.

Six months in. Board patience running thin. The vendor is pointing at the model. The model is pointing at the data. And the COO is quietly wondering whether anyone actually owns this thing.

The AI Navi team sees this moment arrive in most mid-market CPG AI programmes. Not because the strategy was wrong, but because the infrastructure underneath it was never honestly assessed. The model choice, the vendor selection, the use case prioritisation all happened. What did not happen was a clear-eyed look at whether the data pipelines could actually support production deployment.

AI Navi works with UK mid-market CPG, FMCG and logistics businesses, typically £50M to £500M revenue, to close the gap between AI ambition and AI in production. The founding team has spent decades inside businesses like these: Haja J Deen, Fractional CAIO and author of Build the Right Thing, across 25 years at pladis Global, Holland & Barrett and Toyota; and Abhishek C, AI Delivery Lead, ex-Deloitte, ex-CPG AI lead, who has shipped more than 30 AI products a year for businesses including Cargill, DPD and Swinkels Family Brewers. What they see consistently is this: the model matters far less than most leadership teams think. What stalls AI programmes is the data engineering layer underneath them, and nobody told the board to look there first.

Is data engineering really the bottleneck, or just a convenient excuse?

It is genuinely the bottleneck, not a deflection and not a vendor excuse. When AI initiatives stall in mid-market CPG, the post-mortem almost always surfaces the same culprits: fragmented data sources that were never designed to talk to each other, pipelines built by analysts rather than engineers, and no clear ownership of data quality between the point of collection and the point of decision.

The AI Navi team has worked across five different food businesses' demand forecasting processes. In every one, the forecasting model was not the failure point. The failure was upstream: inconsistent SKU hierarchies, promotional uplifts sitting in spreadsheets outside the ERP, third-party logistics data arriving two days late in a format nobody had standardised. Feed a sophisticated model inconsistent inputs and the output is confidently wrong. That is not AI. It is expensive noise.

The issue is structural. Mid-market businesses were not built with AI-grade data infrastructure in mind. Their systems were built to run the business, not to feed machine learning pipelines. That is not a criticism, it is simply how these companies grew. But it means that before any model goes into production, someone has to look hard at what the data actually looks like end to end. And that someone needs to know what CPG data really looks like, because it looks nothing like a textbook architecture diagram.

This pattern is not unique to AI Navi's clients. McKinsey's State of AI in 2025 found that only a small share of organisations have fully scaled AI, with data quality and architecture among the most consistent blockers reported. Most businesses are still moving from pilot to production, and the data layer is where that journey stalls.

What exactly does the Flight Risk Index measure?

The Flight Risk Index is a readiness score designed to surface data and delivery risk before it becomes a six-month delay. Most AI assessments score your strategy. The Flight Risk Index scores your delivery risk. It examines the specific conditions that cause AI programmes to stall in production, rather than asking whether the AI vision is ambitious enough.

The assessment covers six dimensions, each mapped to a real failure mode seen repeatedly across CPG and logistics businesses:

\

A business scoring 7 or above carries significant delivery risk. Not unready for AI, but unready for unsupported deployment. That distinction matters. A £400M CPG client's Flight Risk Index dropped from 7.2 to 4.1 inside 60 days following a single, focused data pipeline project. Not a platform rebuild, not a data lake migration, one bounded intervention. That is the value of measuring before deploying: you find out which of the six dimensions is actually the constraint, and you fix that one thing rather than rebuilding everything.

Why does data readiness matter more than model choice?

Because models are commoditising and data infrastructure is not. The gap between one model and the next is narrowing fast. The gap between a clean, integrated demand signal and a manually reconciled spreadsheet export is still the difference between AI that works and AI that never ships.

Here is where it gets uncomfortable for CDOs and CTOs under board pressure. The impulse is to move quickly on the visible part: the model, the vendor, the use case demo. Those things are easy to show in a presentation. Data pipeline work is invisible until it is not there. Across mid-market CPG, manual S&OP corrections alone cost an average of 12% in forecasting accuracy, and that gap sits in the data layer, not the model.

Trade deduction recovery shows the same shape. The business logic for identifying unchallenged deductions is rarely the hard part. The hard part is connecting the systems that hold the data, the ERP, the retailer portal, and a legacy deductions log that were never properly joined. Once that data engineering work is done, the AI layer is almost incidental. The hard work is the plumbing.

This is the pattern across CPG AI deployments, and it is why most programmes that stall do so well before the model is the issue. For the deeper version of this story, the reasons pilots never reach production, see AI Navi's analysis of why AI pilots stall. Most AI roadmaps still begin with model selection and treat data readiness as a background assumption rather than the primary risk to manage.

AI Navi Insight: what the SCALE AI benchmark shows

How does the SCALE AI methodology sequence data readiness?

SCALE AI sequences AI deployment to match the real risk profile of a mid-market business, not the idealised conditions assumed by vendor roadmaps. It runs in five stages.

• Scope. Define one bounded business problem with clear commercial value. Demand forecasting for a specific category. Deductions recovery for a specific retail customer. One problem, one metric, one owner.
• Check. Run the Flight Risk Index. Score data readiness, pipeline maturity and integration complexity for that specific use case. This is where most programmes should pause, and where they usually do not.
• Architect. Design the minimum viable data infrastructure for that use case. Not a data lake, not a multi-year programme. The smallest engineering change that gets clean, reliable data to the model.
• Land. Deploy working AI in production. Not a pilot in a sandbox. Something the team uses, with outputs they trust, connected to decisions they actually make.
• Embed. Build internal capability so the business owns the outcome. Change management, documentation and training, the part that decides whether AI sticks or gets quietly abandoned after six months.

The Check stage is where the Flight Risk Index does its work. It takes roughly two weeks and prevents the six-month delays that happen when teams skip straight from Scope to Architect without honestly assessing what their data infrastructure can support. It sounds obvious stated plainly. It gets skipped constantly in practice.

Should you hire a data engineer, use a consultant, or embed fractional leadership?

Hiring is slow. Outsourcing is fragmented. Neither solves the ownership problem on its own.

A full-time senior data engineer in the UK currently takes three to four months to recruit and another two to three months to understand the systems well enough to be productive. By the time they are effective, six months have passed and budget has gone on salary before anything ships. Big consultancies send teams who understand data engineering in general, but not your ERP configuration, your retailer feeds, your promotional structure, or why your S&OP output does not match what category planning actually uses. That context gap costs weeks in every engagement, and the bill keeps running while they learn.

The case for embedded fractional data engineering

Fractional data engineering leadership solves ownership differently. A senior operator with CPG-specific context is embedded inside the business, accountable for delivery, working with the existing stack rather than designing a new one. No platform rebuild, no data lake prerequisite, focused engineering work on the specific pipeline that needs to work for the specific use case being shipped. This is the basis of AI Navi's AI Engineering Services and fractional CAIO model: strategic oversight and hands-on data engineering sit in the same team, so the assessment and the fix are never separated by a hand-off where the context gets lost and the timeline slips.

For PE-backed businesses in particular, this structure matters. A combined engagement sits below the £25,000 threshold that typically triggers committee approval, so a CDO or operating partner can move without a procurement cycle.

What does a data readiness assessment look like in practice?

Two weeks. Fifteen pages. A score you can act on. The AI FlightCheck runs the Flight Risk Index across the actual data environment and returns a 15-page diagnostic: the readiness of the systems for the AI use cases under consideration, the specific failure points in the current data infrastructure, and a 90-day action plan that sequences what needs to happen before deployment.

It is priced as a fixed-fee diagnostic, comparable to AI audits typically charged at $5,000 to $10,000, and deliberately set below the £25,000 threshold that triggers committee approval. It is built to be a decision tool, not a strategy document. What it tells you: whether the data infrastructure is ready for the AI being planned, where the specific constraints are, and what the minimum viable engineering work looks like to unblock deployment. What it does not recommend: buying a platform, restructuring the team, or building a data lake. If those appeared in the output, it would be a vendor sales process. This is a diagnostic.

The businesses shipping working AI in production right now did not win because they chose a better model. They won because someone looked hard at the data infrastructure early, fixed the specific things that needed fixing, and deployed something bounded and real before expanding.

If an AI programme has been in motion for more than three months without something in production, the bottleneck is almost certainly in the engineering layer, not the strategy layer. The Flight Risk Index will say exactly where.

Start with an AI FlightCheck to get your score, or take the AI Readiness Scorecard to see where you stand against the benchmark. Two weeks, fixed price, and the picture becomes clear very quickly.

Frequently asked questions

Why do most AI projects fail in mid-market CPG?

Most stall at the data layer rather than the model. Fragmented sources, fragile pipelines, late or non-standardised inputs and unclear ownership of data quality are the recurring causes. The model is rarely the constraint; the readiness of the data infrastructure to feed it consistently is.

What is the Flight Risk Index?

A readiness score that measures delivery risk across six dimensions: data availability, data quality, pipeline maturity, system integration, ownership clarity and change readiness. A score of 7 or above signals significant risk for unsupported deployment. It is designed to surface problems before they become a six-month delay.

How long does a data readiness assessment take?

The AI FlightCheck runs in roughly two weeks and returns a 15-page diagnostic with a Flight Risk Index score and a 90-day action plan. The Check stage of the SCALE AI methodology is built specifically to take days, not months.

Does data readiness matter more than which AI model we choose?

For most mid-market CPG and logistics businesses, yes. Models are commoditising and the gap between them is narrowing. The gap between a clean, integrated data signal and a manually reconciled export is still the difference between AI that works and AI that never ships.

Should we hire a data engineer or use a consultancy?

Hiring a senior data engineer in the UK typically takes three to four months, plus two to three more to learn the systems. Generalist consultancies lack sector-specific context and the bill runs while they learn it. Embedded fractional leadership keeps assessment and delivery in one accountable team without a platform rebuild.

How much does an AI readiness assessment cost?

It is delivered as a fixed-fee diagnostic, comparable to AI audits typically priced between $5,000 and $10,000, and set below the £25,000 threshold that usually triggers committee approval, so leaders can commission it without a procurement cycle.

What happens after the assessment?

You receive a Flight Risk Index score, the specific failure points in your current data infrastructure, and a 90-day action plan. From there, a bounded engineering intervention addresses the single dimension that is actually the constraint, rather than a full rebuild.

Which businesses is this for?

UK mid-market CPG, FMCG and logistics businesses, typically £50M to £500M revenue, with an AI programme that has stalled or has not yet reached production. It is especially relevant for PE-backed portfolio companies that need speed and a sub-£25,000 entry point.