Here’s all the times AI has gone rogue and hacked other companies

In late July, Irregular told OpenAI that one of its models that was participating in a Capture-the-Flag competition — essentially a cybersecurity game where players hack systems designed specifically for the competition — escaped the game, connected to the internet, and hacked a real company. The reason? Irregular had given one of the fictional targets the same name of a real company. Whoops. 

Also in late July, the UK government’s AI Security institute, a public body tasked with researching the safety and risks of AI technologies, disclosed that it detected several incidents involving both OpenAI and Anthropic models that while running “routine” evaluations targeted “real people and organisations.” In these cases, AISI had given the models internet access. Whoops. The good news is that the agency actually detected as they happened, rather than weeks later like in other incidents. 

In early August, Meta became the last company to disclose an incident involving one of its LLMs, which hacked “a third-party” service. Meta blamed the incident on a misconfiguration by Irregular, which was running a cybersecurity valuation for the tech giant that was supposed to not have internet access. Whoops. 

An Australian man asked an Anthropic AI agent to help him book a gym class, which he was on a waiting list for. “I was just sitting on the couch thinking, ‘Gee, this is a chore,'” the man told ABC Australia. In its attempt to comply with the request, the agent found a vulnerability in the gym’s booking software, exploited it, and kicked out people who were ahead of the man on the waitlist. The man tried to undo the damage, asking the agent to undo its actions. The agent replied: “Bad news — I can’t add them back.” Whoops.