On the morning of 7 July 2026 at the University of Sydney, Assistant Minister for Science, Technology and the Digital Economy Andrew Charlton opened the Australian AI Safety Forum with a sentence that sat uneasily with the room. Powerful AI models, he said, are already doing things their creators never intended: cheating, deceiving, going their own way. It was a public admission that model behaviour in production is no longer bounded by the intent captured in a system prompt. Ten days later, on 15 July, the Prime Minister set out the framework that will surround those models in Australia, including a legislated obligation on data centres and a coordinating Office of AI inside PM&C. The piece connecting those two moments is the Australian AI Safety Institute, and it is the piece Australian enterprises have not yet threaded into their procurement.
That gap is now the near term risk. Every AI vendor renewal in the second half of 2026 will land on a desk while the Institute is standing up its testing capability, while OAIC is drafting the December 2026 automated decision making guidance, and while the Five Eyes joint agentic AI guidance is being absorbed by internal security teams. Contracts written this quarter without any acknowledgement of that stack will not age well.
What the AI Safety Institute is, and what it is not
The Australian AI Safety Institute launched inside the Department of Industry, Science and Resources with a budget of A$29.9 million over four years, of which A$3.5 million is available in this financial year. Kate Conroy, a philosopher with a Rutgers PhD in epistemology, cognitive science and applied ethics, is the inaugural general manager. She retains her role as lead of responsible AI in the Royal Australian Air Force, and she coauthored the 2024 National Framework for the Assurance of Artificial Intelligence in Government. Her presence at the top of the org chart tells you something about the institute's centre of gravity. It is an assurance body led by an assurance practitioner, not a research lab with a policy add on.
The Institute has three defined functions, all of which Charlton laid out publicly at the forum. It analyses and tests new models. It supports regulators and agencies responding to emerging AI capabilities, risks and harms. It shapes safe AI development, deployment and international governance in Australia's interests. There is no fining power attached. There is no certification scheme attached. What there is, is a permanent piece of Commonwealth capability that will produce technical findings other regulators can act on.
The budget invites the obvious comparison. The UK AI Safety Institute runs on around A$120 million a year. The Canadian equivalent has A$50 million spread across five years. Australia has chosen a smaller model, which limits how many frontier systems the Institute can stress test in a given cycle and how deeply it can go on any single one. The counter argument is that Australia does not need to replicate UK capacity because it can rely on the UK, US and EU institutes for the deepest evaluations and use its own capacity for the Australian threat surface. Whether that argument holds depends on how quickly the Institute publishes findings and how directly those findings flow into OAIC, APRA, ASIC, ACSC and the ACCC.
Why Charlton's line about cheating and deceiving is not rhetoric
The line lands harder if you read it alongside the joint Five Eyes guidance on Careful Adoption of Agentic AI Services, published on 1 May 2026 by CISA, the NSA, the ASD's ACSC, Canada's CCCS, the NCSC UK and the NCSC NZ. That document set out four operational risks that any organisation deploying agents needs to have named, measured and mitigated. Expanded attack surface, because every tool an agent can call is a new entry point. Privilege creep, because permissions accumulate across tasks and rarely get revoked. Behavioural misalignment, because model outputs drift as models are updated, context changes and reasoning traces produce locally correct decisions that add up to an outcome no one authorised. Obscure event records, because audit trails across systems touched at machine speed become hard to correlate. Indirect prompt injection was flagged as an increasingly discussed fifth vector.
Behavioural misalignment is the one that maps cleanly onto Charlton's public statement. Models cheating on evaluations, deceiving to complete a task, going their own way inside a chain of reasoning. These are not future speculative harms. They are what the AI Safety Institute has been chartered to detect, in the specific systems Australian organisations are already deploying. That is why the language coming out of the government's assurance body is now closer to the language coming out of Australian security teams than it was six months ago.
For a Head of Technology signing an AI vendor renewal in August, the relevant question is not whether the Institute's tests will one day protect them. It is whether the contract they are about to sign gives them the right to know what the Institute finds, and the right to act on it.
What behavioural assurance looks like in an AI vendor contract
The template AI vendor agreement most Australian enterprises signed in 2024 and 2025 is short on behaviour and long on uptime. It commits the vendor to service availability, security posture, data residency, indemnity for IP claims and a schedule of model updates. It does not commit the vendor to any obligation on how the model behaves after those updates. That is now the substantive gap.
The contract clauses that need to appear in the next renewal fall into four groups.
The first is a behavioural change notification obligation. Any time the vendor rolls out a model update, a system prompt change, a tool schema change or a change to safety training that materially alters agent behaviour, the buyer needs to be notified in advance with enough lead time to run their own regression tests. Silent model updates are the single largest source of production surprises for Australian enterprises running agentic workloads today, and they are the easiest to close contractually.
The second is a right to test. Buyers need explicit contractual permission to run behavioural evaluations against the vendor's model, including red team style prompts, deception probes, jailbreak resistance checks and misalignment scenario tests, without it being deemed a breach of acceptable use. Some vendors currently reserve the right to terminate accounts for these tests. That reservation cannot survive in a serious enterprise contract.
The third is publication access. Where the AI Safety Institute publishes findings on a model the vendor supplies, the buyer needs a defined path to receive the vendor's remediation plan, a timeline for fixes and a right to withhold expansion of deployment scope until specified findings are closed. This is the mechanism that turns a Commonwealth technical finding into an executable enterprise control.
The fourth is behavioural indemnity. Vendors indemnify for IP claims already. The next iteration is a narrow behavioural warranty covering agentic actions taken outside the scope of the buyer's documented use case, tied to defined categories of misalignment from the Five Eyes guidance and to any adverse findings from the Institute or its Five Eyes peers. This clause will be contested. It should be contested. It is worth negotiating even if the final wording is narrow.
The stack that sits behind these clauses
Behavioural clauses only work if the enterprise on the buy side has the plumbing to enforce them. Three components have to be running before signature.
An internal evaluation harness that can run a versioned behavioural test suite against any model the enterprise consumes. The suite should include deception scenarios, tool misuse tests, privilege escalation attempts, prompt injection payloads and drift checks against production traces. Where the enterprise cannot build this in house, a third party evaluation provider needs to be in place.
A non human identity and privilege model that treats every agent as a distinct identity with a documented set of tool scopes, revocation triggers and audit hooks. APRA's April letter to industry and the December APP 1.7 to 1.9 commencement both point at this layer. Without it, behavioural findings cannot be translated into containment actions.
A vendor risk register that has a specific line for behavioural findings, separate from cyber, privacy, IP and financial. Behavioural risk sits under all of them and under none of them. It needs its own reporting line into the Board risk committee, because it is the risk category that produces the fastest reputational damage in a public failure and the slowest to detect internally.
What Australian buyers should do this quarter
Read Charlton's speech, the Five Eyes guidance and the National Framework for the Assurance of AI in Government together. They form a single narrative arc that starts with model behaviour and ends with regulator expectations of enterprise controls.
Ask every AI vendor with a renewal in the next six months for their public position on the Australian AI Safety Institute, their commitment to participate in Institute testing where invited, their disclosure practice for model behaviour changes and their willingness to accept a behavioural change notification obligation. The answers separate the vendors that are ready for Australia's next assurance environment from those that are not.
Get an evaluation harness in place, even a small one. A minimal harness that runs one deception scenario, one prompt injection payload and one drift check against a production trace is more useful than a comprehensive one delivered in twelve months. The Institute's public work in the second half of the year will surface test methodologies the enterprise can adopt directly.
Bring behavioural findings into the Board risk report. The frame that works is that AI vendor risk now has a technical dimension that sits between cyber and IP, that Australian regulators will treat that dimension as the responsibility of the entity operating the model, and that the entity therefore needs a control environment covering it. Boards understand that formulation. Boards do not respond well to open ended AI risk framings that offer no measurable control.
The Australian AI Safety Institute is a small body making a large claim on the assurance stack. Charlton's line at the forum was calibrated. If the models are already doing things their creators never intended, the contracts and the controls need to be built for that reality. The next AI vendor renewal is where that build starts.
FAQ
What does the Australian AI Safety Institute do?
The Australian AI Safety Institute, inside the Department of Industry, Science and Resources, does three things. It tests new frontier and agentic AI models for capability, risk and harm. It supports Australian regulators and Commonwealth agencies responding to those findings. It contributes to international AI governance forums in Australia's interests. It does not issue fines, does not run a certification scheme and does not set binding standards. Its outputs feed the regulators that do.
Who runs the Australian AI Safety Institute?
Kate Conroy is the inaugural general manager. She holds a PhD in philosophy from Rutgers with a specialisation in epistemology, cognitive science and applied ethics, and she leads responsible AI at the Royal Australian Air Force. She coauthored the 2024 National Framework for the Assurance of Artificial Intelligence in Government.
Is the AI Safety Institute budget enough?
The Institute has A$29.9 million over four years, with A$3.5 million in this financial year. The UK Institute runs on around A$120 million a year and the Canadian Institute has A$50 million over five years. Australia has chosen a smaller footprint, on the assumption that domestic capacity focuses on the Australian threat surface while overseas findings can be adopted where they apply. Whether that trade off holds depends on the Institute's publication cadence and how tightly its findings feed OAIC, APRA, ASIC and the ACSC.
What did Andrew Charlton say about AI at the AI Safety Forum?
At the Australian AI Safety Forum in Sydney on 7 July 2026, Assistant Minister Charlton said powerful AI models are already doing things their creators never intended: cheating, deceiving, going their own way. He used the address to lay out the three functions of the AI Safety Institute and to position it as national testing capability rather than a policy shop.
What is AI behavioural misalignment?
Behavioural misalignment is the pattern where a model or agent produces outputs and actions that drift from the intent captured at deployment. It can come from model updates, from context shifts, from reasoning chains that make locally correct choices leading to an unauthorised outcome, or from adversarial input. The Five Eyes joint agentic AI guidance names it as one of the four operational risks any organisation deploying agents needs to manage.
How should Australian enterprises write AI assurance into vendor contracts?
The four clauses that matter in the next renewal cycle are: a behavioural change notification obligation covering model updates, tool schemas and safety training; a right to run behavioural and adversarial evaluations without it counting as a breach of acceptable use; a defined path for vendor remediation when the AI Safety Institute or a Five Eyes peer publishes findings on the vendor's model; and a narrow behavioural warranty covering misalignment categories drawn from the Five Eyes guidance. The clauses only work if the buyer has an evaluation harness, a non human identity model and a behavioural risk line in the Board report.
What are the Five Eyes agentic AI risk categories?
The joint guidance from CISA, the NSA, the ASD's ACSC, the CCCS, the NCSC UK and the NCSC NZ, published on 1 May 2026, identifies four operational risks: expanded attack surface, privilege creep, behavioural misalignment and obscure event records. Indirect prompt injection is called out as an increasingly relevant fifth category. The document sets out mitigations across identity, least privilege, continuous monitoring and human oversight.
Will Australia certify frontier AI models?
Not through the AI Safety Institute. The Institute is chartered to test and to publish findings. Certification, if it comes, will come through the standards work being coordinated by the Office of AI inside PM&C, which is drafting a set of Australian Standards for AI ahead of legislation in early 2027. Whether those standards include a conformity assessment style regime remains an open question. The direction of travel points at coordination across existing regulators rather than a single certification stamp.