Inside OpenAI’s plan to make ChatGPT the way you use your computer

OpenAI staffers say they’re close to realizing one of the AI lab’s longest-standing ambitions.

Since the company’s 2015 founding, leaders have hoped to one day build AI agents that can use a computer — and, crucially, a web browser — with the same dexterity as a human.

A huge challenge has been collecting and creating the data to train these agents.

OpenAI now has enough faith in the tech to start rolling out Computer Use tools to customers.

This year, the company launched a Chrome extension that lets ChatGPT take over the browser, created cloud and in-app browsers for ChatGPT to interact with public websites, and added Computer Use to its Codex coding tool.

The same underlying tech now also lets users have ChatGPT complete tasks on other apps.

OpenAI’s president, Greg Brockman, wrote on X that an April update made his company’s tech “no longer just for coders, but for anyone who does computer work.” Employees who work on the tools for OpenAI told Business Insider that while there’s room for improvement, the tools are at an inflection point.

Rival Anthropic is also racing to improve its version of Computer Use and was the first to market in 2024.

As of May, at least 600,000 organizations had tried Anthropic’s Claude Cowork feature, which uses the tool.

Now, OpenAI says its tech’s capabilities have caught up, and it’s filling its blockbuster ChatGPT with ways to automate computer tasks, betting that users will turn over more and more work to it. “Once ChatGPT can use computers and software faster than you or I can, it’s going to change the way that you, by default, want to interact with your computer,” Ari Weinstein, a manager on OpenAI’s Computer Use team, told Business Insider.

Each phase of training for OpenAI’s newest models included data or processes that improved them for Computer Use.

Zhou Yu, an AI researcher at Columbia University who worked on a popular benchmark test for computer-use agents, told Business Insider that this type of extra training builds on a standard AI model’s capabilities and teaches it how to handle more tasks.

At the earliest phase, where OpenAI builds a model’s base with a vast amount of data, the company likely included frame-by-frame screenshots, Yu said, to “front-load” helpful information for Computer Use.

Then, in a later step, developers can provide the models with data showing the ideal inputs that produce a given output — in this case, the computer functions that lead to that result, such as filling out a tax form or 3D-modeling an object.

This data can come from human trainers: OpenAI, when it announced a 2025 version of Computer Use, said it had used datasets where people demonstrated how to complete tasks. (OpenAI declined to provide specifics about its latest training data.) Yu said this type of data is expensive since it’s hard to obtain and clean up for use.

OpenAI also likely improves the models’ computer-use skills by ordering them to try to complete virtual tasks and rewarding successful behavior — known as reinforcement learning, where the model “can learn to do more of these actions that have gotten rewarded,” Yu said.

OpenAI has upgraded how its models interact with a computer and process data while in use, Weinstein said.

Previously, ChatGPT would take a screenshot of a user’s computer, analyze pixels, inject a command, and repeat.

Now, when the tool has a website open, it can swiftly read the page’s guts — its memory structure, accessibility information, and link connections.

The tool still takes screenshots; users are asked to let ChatGPT record their screen as they set up Computer Use, so it can rapidly analyze what’s on-screen.

Weinstein said the mix of tools helps provide OpenAI’s Codex with a way to test the code it’s built, find bugs, and make improvements in a “full loop.” That’s the use case for Cristian Medina Ruiz, a hobbyist coder in the Czech Republic.

He showed Business Insider his attempt to use Codex to rebuild the 2013 city-building game “SimCity” from its original source code.

The Computer Use tool opened a window on his computer to verify that the game correctly simulated car movements and building images.

Ruiz said Computer Use got him wholeheartedly into ChatGPT, and he now leaves his code running and iterating when he’s away from his computer. “It does these things even when I’m away,” he said.

Weinstein and James Sun, who works on browser capabilities at OpenAI, told Business Insider that part of the technology’s promise is connecting ChatGPT’s agents to disparate parts of the internet and its “long tail” of websites.

Most online tools are built only for the human hand — an appointment-scheduling website, a company’s internal inventory dashboard, a social media platform.

Weinstein and Sun said they’re seeing users of Computer Use automate data entry work, compliance tasks, and calendar scheduling.

Weinstein said that as OpenAI’s models have improved over time, they’ve become better at noticing when things go wrong and correcting them, which helps them finish tasks and avoid going down online rabbit holes.

Yu, the Columbia AI researcher, said there’s a “gap” between the technology’s current speed and the level consumers want. “We definitely do see that computer-use agents could do well in limited domains, when we give it a lot of training data,” Yu said. “When you want to generalize to any webpage, any application, any operating system, it’s still very difficult.” Computer Use can’t churn through email responses, and scrolls laggingly through social media feeds.

Both Sun and Weinstein hope that as OpenAI’s models improve, Computer Use will become so fast that the experience of working online will get the same treatment as coding. “For a lot of engineers now, it’s just faster for the agent to code than to code themselves,” Sun said. “There’s no calculus, you just do it.

I think Computer Use will become more and more like that.” When OpenAI CEO Sam Altman was discussing an early Computer Use version in 2025, he wrote on X that, “Although the utility is significant, so are the potential risks,” and added, “We recommend giving agents the minimum access required to complete a task to reduce privacy and security risks.” The fact that OpenAI puts this onus on users worries Mark Beare, a general manager at the security firm Malwarebytes. “This does still feel a little bit wild west-ish to me,” Beare told Business Insider, adding that customers might not have enough privacy protections set up and that businesses also struggle to keep pace with AI governance.

Beare added that people trying Computer Use should make sure they wall off passwords, sensitive data, and anything under a nondisclosure agreement or protected by privacy law.

He said he’d avoid letting an agent run unsupervised on a long, complex task — he worries that a bad actor’s website will lure in the bot and try to extract personal information.

Sun said his team has been grappling with what he called “confirmation policy,” or how often and when the Computer Use tool should ask a user to permit its next step.

So far, he said, they’ve decided that whenever an AI agent transmits data or deletes something, it should ask the user first.

Much of doing work online requires quick decisions — Sun said OpenAI is still figuring out the trade-offs of letting AI onto browsers. “You can make something 100% safe, but then it’s just very, very difficult to use, and you can make something that is very easy to use but very dangerous,” Sun said. “So we’re really trying to figure out how to build something that’s beneficial and also easy to use.” Have a tip?

Contact this reporter via email at scouncil@businessinsider.com, or over text, Signal, Telegram, or WhatsApp at 415-757-8198.

Use a personal email address, a nonwork WiFi network, and a nonwork device; here’s our guide to sharing information securely.

Advertisement