Anthropic’s Claude AI Can Now Control Your Computer Without Any Help

Oct. 23, 2024

Anthropic has updated itsClaude 3.5 SonnetAI model and along with that, it has brought the ability to carry out tasks on the computer for you. The new feature, called a very obvious ‘computer use’, is designed tomimic human actionson a computer like clicking, typing, and moving the cursor. Anthropic Claude AI’s computer use isavailable on the APIfor developers, starting today in beta.

In theofficialrelease post for the feature, Anthropic gives us a good look at how the feature works. We see Sam Ringer, an Anthropic Researcher, testing out the tool. He prompts Claude to fill out a request form by extracting information from the spreadsheet and online portal that it’s seeing on the screen.

We then see the tool get to work by screenshotting the information on the screen. Additionally, we see the tool search from an online portal (as directed) to look for information that was unavailable on the spreadsheet. Side-by-side, Claude opened up the form and started filling it up.

Anthropic notes that Claude 3.5 Sonnet scored14.9% in the screenshot-only categorywhich is “notably better than the next-best AI system’s score of 7.8%.” Additionally, Anthropic states that, “when afforded more steps to complete the task, Claude scored 22.0%.”

Developers can further build on the computer use feature using Anthropic API, Vertex AI as well as Amazon Bedrock. Anthropic also notes that the updated Claude 3.5 Sonnet shows a range of improvements on industry-leading benchmarks.

Additionally, Anthropic has also released a newClaude 3.5 Haikusmaller model which “improves across every skill set and surpasses even Claude 3 Opus.” Claude 3.5 Haiku will be made available on the API later this month.

Teaching general computer skills to a computer certainly sounds interesting, and something I’m looking forward to. I mean, it has the potential to save a lot of time by automating tasks. What do you think about Claude AI’s computer use feature? Drop your thoughts in the comments down below!

Sagnik is a tech aficionado who can never say “no” to dipping his toes into unknown waters of tech or reviewing the latest gadgets. He is also a hardcore gamer, having played everything from Snake Xenzia to Dead Space Remake.