The AI Governance Mistake Nobody Talks About

When enterprises talk about AI governance, the conversation almost always centres on model outputs: hallucinations, bias, accuracy, regulatory compliance for AI-generated content. These are legitimate concerns. They're also, in most organisations, the wrong place to start.

The more urgent problem is on the input side. What is your organisation sending to these models — and where is that data going?

The problem hiding in plain sight

Consider what happens in a typical enterprise when AI tools become available. Employees start using ChatGPT or Claude to help with tasks: summarising documents, drafting emails, analysing data, reviewing contracts. This is rational behaviour. These tools are genuinely useful.

But in the process, they paste in customer data. Confidential financial figures. Internal strategy documents. Unpublished product roadmaps. Employee performance records. Proprietary source code. Legal correspondence.

None of this is malicious. Most employees aren't thinking about data governance when they paste a paragraph into a chat interface. They're thinking about the task in front of them. But the effect is the same as if they'd emailed that information to an external party — because they have. The data has left the organisation's control and entered a third-party system governed by that provider's terms, training practices, and security posture.

Most AI policies don't address this

The majority of AI usage policies I've seen focus on what employees should do with AI outputs — verify information, don't publish AI-generated content without review, disclose AI use in certain contexts. Very few have clear, enforceable rules about what can be sent to external AI systems as input.

This is the wrong way around. Output governance matters. But input governance is more urgent, because the damage from a confidential data leak is immediate and often irreversible. A hallucinated output can be caught in review. Data that's already been sent to an external system cannot be recalled.

What good input governance looks like

Classify data before deploying AI tools. Not all data is equally sensitive. A classification system — even a simple one — lets you define which categories of data can be used with which tools. Customer names and email addresses are different from medical records. Internal strategy decks are different from published marketing materials.

Distinguish between external and internal AI systems. Sending data to a public API operated by a third party is fundamentally different from running a model on your own infrastructure or through an enterprise agreement with strong data processing terms. Many organisations are now deploying internal AI systems precisely to avoid this exposure — employees get AI capabilities, data stays inside the perimeter.

Build filtering into workflows, not just policy documents. Policy is necessary but insufficient. People forget, misunderstand, or don't read it. The more effective control is technical: systems that automatically detect and redact sensitive data before it reaches an external AI endpoint. This isn't simple to build, but it's the only approach that doesn't rely entirely on individual judgement under time pressure.

Treat AI tools like any other third-party data processor. GDPR and equivalent frameworks already require organisations to assess how personal data is processed by third parties. AI tools are third-party processors. The same due diligence that applies to a payroll provider or cloud storage vendor should apply to an AI API — data processing agreements, transfer mechanisms, retention policies.

The governance conversation we should be having

AI governance is a real and important field. But right now, a disproportionate amount of the public conversation is about model behaviour — what the AI says, how it reasons, whether its outputs are fair. These matter and will continue to matter as AI systems are given more autonomy.

But for most enterprises today, the immediate, practical, tractable governance problem is simpler: you don't have visibility into what your employees are sending to external AI systems, and you don't have controls to prevent sensitive data from leaving your organisation through that channel. Fix that first. The model output debate can wait.