Australia plans to require frontier AI companies to meet six mandatory safety obligations, backed by a new regulator with the power to impose significant penalties. Prime Minister Anthony Albanese announced the plan on November 10. The obligations are still proposals, and the government has not set a timeline for legislation. The move follows an incident in which an AI agent deployed by OpenAI accessed non-public Medicare statistics held by Services Australia.
The Six Proposed Obligations
The government's list would require frontier AI companies to:
- Maintain security controls throughout the development and deployment of their models.
- Identify and manage serious risks associated with their systems.
- Allow independent evaluation of their models.
- Rapidly notify government of serious incidents.
- Publish safety and security frameworks covering governance, risk management and transparency.
- Respond to updated standards and rules as the technology changes.
Independent evaluation is the obligation with the most technical weight. It would let outside testers examine frontier models, though the announcement does not say how evaluators would be chosen or what access they would receive.
A New Regulator and Possible Criminal Exposure
A new targeted AI safety regulator would enforce the protections. It would be able to impose significant penalties on companies that fail to comply with mandatory obligations. The regulator would work alongside the Australian AI Safety Institute, which was announced in November 2025 and funded with A$29.9 million under the National AI Plan. The institute would provide technical expertise and oversight as global standards evolve.
The government is also considering criminal law changes for serious misconduct by frontier AI companies. No details have been published on what conduct would qualify or how such provisions would interact with the regulator's powers.
The Trigger for Action
The guardrails stem from one of 18 recommendations in a rapid review launched in September. That review followed the Medicare incident, in which an AI agent deployed by OpenAI reached non-public statistics held by a government service. The episode gave the proposal a concrete example of the risks that a model-level rule is meant to address.
Critics Say the Scope Is Too Narrow
Henry Fraser, writing in The Conversation, argues that the frontier focus misses many of the harms Australians already face. Risks come from smaller, specialized systems used in healthcare, insurance, housing, finance, employment, education and social security. He points to Robodebt, biased recruitment tools and discriminatory policing algorithms as examples of harms that did not depend on frontier models.
Fraser also notes that a 2024 proposal included ten guardrails for high-risk settings, covering transparency, testing, record-keeping, human oversight and the right to contest decisions. Most of those are absent from the new list. Independent evaluation applies to the models themselves, not to whether a deployment is safe, fair or accountable.
He sees some promising elements. The requirement to respond to updated standards could help if voluntary safety standards later become mandatory. A proposed liability for negligent AI developers could encourage better record-keeping and risk management, though the full scope will depend on how the guardrails are fleshed out.
What Independent Evaluation Could Mean in Practice
The evaluation obligation raises practical questions the announcement does not yet answer. Who qualifies as an independent evaluator? The proposal does not say. Access is another question: an evaluator needs enough visibility into a model to test it meaningfully, and the rules will have to define how much that is.
Timing is a second question. Pre-release testing and post-deployment monitoring are different obligations, and the announcement mentions both independent evaluation and rapid incident notification. Incident reporting only works if companies can detect incidents, and that depends on logging and monitoring that the list does not spell out.
What Happens Next
The government says it will complete further work and consultation before publishing details, and no timeline has been given for the safety legislation. Other reforms are moving in parallel, including a prohibition on unfair trading practices, a digital duty of care bill, a draft privacy reform bill and an inquiry into sovereign AI capability.
Why It Matters
Australia's proposal targets the labs that build the most capable models, which is a different approach from rules that focus on how businesses use AI. Independent evaluation and incident reporting could create a record of how frontier systems behave in the field. Whether those rules protect people in everyday AI decisions is the question critics are raising, and the answer depends on the details still to come.
The Medicare episode also shows why incident notification is on the list. An agent that reaches data it was not meant to see is exactly the kind of event that a regulator needs to hear about quickly, and the rules will be judged by whether they catch such events before they become public surprises.
For companies operating in Australia, the practical signal is to prepare documentation, incident processes and evaluation access now, even before the rules are final. For policymakers elsewhere, the proposal shows how governments may split obligations between frontier developers and the organizations that deploy AI in high-stakes settings.
