AI Governance Says Test Adversarially. Nobody Wrote Down What That Means.

01 One sentence, three documents

Read that twice. It establishes that a test must happen. It does not say what the test covers, which failure modes count, how deep it goes, or what result lets you ship. Provision 9.2 adds that "System Operators shall conduct testing prior to the system being deployed with support from Developers". Provision 9.2.1 adds that both System Operators and Developers "should use independent security testers with technical skills relevant to their AI systems". The words red teaming and penetration testing appear nowhere in the document.

Then the sentence travelled. DSIT said it would submit the Code to the European Telecommunications Standards Institute [ DSIT publication page ] . It did. In April 2025 the Code became ETSI TS 104 223, where provisions 5.2.5-1 through 5.2.5-4.1 carry the Principle 9 text word for word [ ETSI TS 104 223 ] . In December 2025 ETSI elevated it again, to European Standard EN 304 223 V2.1.1. Provision 5.2.5-1 is unchanged [ ETSI EN 304 223 ] .

02 What the frameworks actually say

The NCSC got there first. Its Guidelines for secure AI system development, published 27 November 2023 with international co-signatories, tell you to release models "only after subjecting them to appropriate and effective security evaluation such as benchmarking and red teaming" [ NCSC 2023 ] . That is the only mention of red teaming in the guidelines. It describes an outcome. The further reading points at open-source security testing libraries instead of defining what the evaluation must cover.

ETSI carries the one explicit adversarial obligation. Provision 5.1.2-2 states that developers and system operators "shall ensure that AI systems are designed and implemented to withstand adversarial AI attacks, unexpected inputs and AI system failure" [ ETSI TS 104 223 ] . The standard defines the term adversarial attack in its terminology section. It gives no procedure for demonstrating that a system withstands one. Withstand is a verb with no measurement attached.

DSIT has already named the consequence. Its assurance market study counts an estimated 524 firms supplying AI assurance goods and services in the UK, including 84 specialised AI assurance companies, generating around £1.01bn and employing an estimated 12,572 people, with the potential to exceed £6.53bn by 2035. The same report states that "the use of differing frameworks and terminology between different sectors and jurisdictions has also fragmented the AI governance landscape", and that limited access to information about emerging AI models restricts the supply of third-party assurance [ DSIT assurance 2024 ] . A billion-pound market already exists to answer a question the frameworks still have not asked precisely.

This is why buying the right test matters more than buying a test. A standard penetration test checks the infrastructure around the model and says nothing about prompt injection. AI red teaming attacks the deployed system the way an adversary meets it: the prompts, the retrieval pipeline, the tools it can call, the permissions it holds. We broke down which of those Principle 9 actually expects in our read of the UK Code of Practice.

03 What the test finds when you run it

The UK government does define the test, for itself. The AI Security Institute has evaluated more than 30 frontier systems since November 2023 [ AISI key findings ] . Its December 2025 Frontier AI Trends Report states the result in one line: "We've discovered universal jailbreaks for every single system we've tested to date" [ AISI Frontier AI Trends Report ] . Every system. Including systems whose vendors had already run their own evaluations and published the results.

The attacker side moves faster than any annual review cycle. AISI reports that in late 2023 models could complete apprentice-level cyber tasks 9% of the time, and that today the figure is 50%. In 2025 AISI tested the first model that could complete cyber tasks intended for experts with over ten years of experience [ AISI key findings ] . AISI is also honest about the limit of its own method, noting that model performance in controlled, task-based settings "may not reflect or generalise to real world effectiveness" [ AISI Frontier AI Trends Report ] . Quote that limit to anyone who waves the numbers away. It is the reason you test your system rather than reading someone else's benchmark.

Keep the scope honest. These findings concern frontier model safeguards, not your customer service assistant. AISI has not tested your deployment and neither has Bengio's panel. What transfers is the direction of travel and the base rate. Test the best-resourced engineering teams on earth and you find something. Every time. Your AI security posture is not the exception.

04 The gap between policy and evidence

DSIT measures the gap its own Code was written to close. The Cyber security breaches survey published on 30 April 2026 found that "around a third of businesses (31%) and a quarter of charities (25%) were either using AI, in the process of adopting it or actively considering using it". Of that group, "around a quarter of businesses (24%) and charities (27%) reported having cyber security practices or processes in place to manage the risks from the use of AI technology" [ DSIT breaches survey 2025/2026 ] .

Read that carefully, because it is weaker than it looks and stronger than it sounds. The survey counts one thing: whether any cyber security practice or process exists at all for AI use. It never asks about adversarial testing. Roughly three in four UK organisations touching AI have none. The testing question sits a long way behind that starting line.

The regulated sector shows the same shape at higher stakes. The Bank of England and FCA survey of AI in UK financial services found that 75% of firms already use AI, with a further 10% planning to within three years. In the same survey, "46% of respondent firms reported having only 'partial understanding' of the AI technologies they use versus 34% of firms that said they have 'complete understanding'", and a third of all AI use cases are third-party implementations [ Bank of England and FCA 2024 ] .

05 "At least better than random"

One UK regulator has already opened the box and written down what it found. Between August 2023 and May 2024 the ICO ran consensual audits of developers and providers of AI tools used in recruitment. Across those engagements, "ICO auditors made 296 recommendations and 42 advisory notes". Of the recommendations, 97% were accepted with actions set, 3% were partially accepted, and "no recommendations were rejected" [ ICO audit outcomes 2024 ] .

Zero rejections is the number to sit with. Every one of those organisations had a governance position before the auditors arrived. They had policies. They had assurances from their suppliers. Then independent auditors looked, made 296 recommendations, and the organisations accepted essentially all of them. The audit is what converted a stated position into a factual one.

Two scope points, so you use this correctly. The ICO audits were consensual, covered recruitment tools only, and explicitly excluded generative AI and biometric processing. They are not LLM security findings. What they demonstrate is the mechanism: a written governance position survives right up until an independent party tests it, and then it either holds or it does not.

06 Turning your policy into evidence

You will not get a definition of the test from DSIT, ETSI or the NCSC. They have had nineteen months and three publications to write one. So define it yourself, run it, and keep the result. The frameworks accept evidence. They simply never told you how to produce it.

Here is the specific outcome on offer. Book a free audit and we map every AI system you have exposed to the internet, name the one an attacker reaches first, and show you the path in. It takes a short call and costs nothing. If you want the full picture, we then red-team that system and hand you the working attack paths, the guardrails that close them, and a dated report your governance file can rest on. Your red team. Within reach.

Start at the free audit. Bring one AI system. You will know within a week whether your policy describes reality.