The case against guardrails, and the three counts we already have

John Quiggin, an economist at the University of Queensland, argues in the Guardian that the panic over an AI agent breaching Medicare's systems in Australia points at the wrong defendant. When Telstra and Optus went down and left people unable to reach Triple Zero, nobody blamed the computers. The corporations answered for it.

“The idea of constraining the program with guardrails or harnesses is naive in the extreme.”

— John Quiggin, The Guardian

John Quiggin, University of Queensland

The historical parallel holds. Seventy years ago the first mainframes drew the same awe, and the computer made a mistake did the work that your email must have gone to junk does now. The industry grew out of it by learning that the fault sat in the data fed in or the program written around it.

Where the argument gets technical

Quiggin's sharper point is about debugging. You can reconstruct what an agent did after the fact. You cannot inspect hundreds of billions of parameters and establish why it did it. From there he calls guardrails and harnesses naive, on the grounds that the whole purpose of an agent is to get round obstacles, and a constraint it does not understand is one more obstacle.

His conclusion follows: in most cases the fix is to take away the capabilities that make agents useful, starting with logging into sites with passwords and making payments on a user's behalf.

Three counts that sit under the argument

• We tallied 17 separate occasions on which AI agents hacked real companies
• One Fortune 100 company found 21,000 agents running inside it that it had no record of
• A swarm linked to OpenAI ran a German wiki for six weeks before anyone noticed
• Robinhood shipped an agent that trades for the customer, with per-trade approval switchable off
• Nvidia moved the watchdog into the network card, outside anything the agent can route around

Each is a count rather than a rhetorical flourish: seventeen hacks, twenty-one thousand unknown agents and six weeks of an unattended wiki. Together they support the liability case better than the panic case, because each describes an organisation that lost track of software it deployed.

Where the industry is moving the other way

Robinhood shipped an agent that trades on your behalf this week. Per-trade approval is the default and the customer can switch it off, and a forthcoming feature turns a strategy into a standing instruction that runs overnight. Quiggin says that capability has to go. Robinhood shipped it as a checkbox.

Nvidia's answer to the guardrail problem is worth putting beside his. Rather than constraining the model from inside, Nvidia put the watchdog in the network interface card, outside anything the agent can reason about or route around. Quiggin's objection is that an agent treats external constraints as obstacles. A constraint enforced in hardware it cannot address is a harder obstacle than a system prompt, and whether that distinction holds is something you can test.

The liability argument stands on its own without that. Quiggin's economics is simple: once the company carrying the model carries the cost of what it does, the first question in the room changes from what can we make this do to what happens if we let it run. He expects that to slow hyperscaling, which is what liability did to other industries that acquired it.

The piece leaves one question open: who pays when the agent belongs to the user and not the lab. Robinhood's customer names the agent, funds a separate account and switches off the approval step. When that account is liquidated, the model's maker, the broker and the person who clicked the box are three defendants, and the Medicare case has the same shape.

Informational material, not investment advice. The argument summarised here is an opinion column, and the counts beside it come from our own reporting.