Build a Computer Use Agent on GitHub with Cua AI in Minutes

Cua is a revolutionary open-source framework that empowers AI to operate a computer just like a human—seeing the screen, clicking buttons, and typing. For developers, this opens up a new frontier of automation. However, giving an AI agent free rein on a host machine is risky. Cua solves this by providing safe, sandboxed environments (containers) where agents can run without any chance of breaking the host computer's operating system.

The excitement around this technology is palpable, with the project recently trending #1 on GitHub [1]. This article will guide you through building your first computer use agent on GitHub with Cua in just a few minutes.

What is Cua AI and Why is it a Game-Changer?

At its core, Cua is a framework for building, testing, and deploying AI agents that can control computer GUIs across Windows, macOS, and Linux. Unlike traditional API-based automation, Cua agents interact with applications visually. This means you can automate any task you could perform with a mouse and keyboard, even in applications that don't offer an API.

Safety Through Sandboxing

The most critical feature of Cua is its emphasis on safety. Agents run in isolated virtual machines or containers, completely separate from your main operating system. We've heard from users who, before Cua, had experimental agents accidentally break their OS by doing things like preventing disk writing. Cua's sandboxing entirely mitigates this risk, providing a secure playground for development and execution.

Open-Source and Community-Driven

The entire Cua infrastructure, including its SDKs and Docker images, is open-source and available on GitHub. The project has garnered incredible community support, with over 11.1k stars on its repository, demonstrating a shared excitement for the future of AI-driven computer automation [2].

Getting Started: Your First Cua Agent in 4 Steps

Cua is designed for rapid development. This section walks you through the entire process, from setting up a secure environment to running a live agent. With our cloud option, you can get started without any complex local configuration.

Step 1: Set Up Your Computer Environment

The "Computer" is the sandboxed operating system your agent will control. Cua provides several ways to get started, catering to different needs and operating systems.

  • Cua Cloud Sandbox (Recommended): The easiest and safest way to begin. We provide a pre-configured Ubuntu container that's ready to go in seconds.
  • Local Docker Container: Run a Linux desktop environment locally on any OS that supports Docker.
  • macOS via Lume: Run a full macOS virtual machine on your Mac host.
  • Windows Sandbox: For users on Windows 10/11 Pro, you can use the built-in Windows Sandbox feature.

For detailed setup instructions on each of these options, refer to our Quickstart (CLI) | Cua Docs.

Step 2: Install the Cua SDK

Next, you'll install the Cua Python packages. This single command gives you everything you need to create the agent and the computer environment it will control.

bash
pip install -U "cua-computer[all]" "cua-agent[all]"

This command installs all the necessary libraries to define and run your agent programmatically.

Step 3: Write the Agent Code

Controlling the agent requires just a few lines of Python. In the script below, we initialize a computer environment, create an agent with a powerful vision model, and assign it a simple task.

python
from agent import ComputerAgent
from computer import Computer

# Initialize a computer (e.g., a local Docker container)
computer = Computer(
    os_type="linux",
    provider_type="docker",
    name="trycua/cua-ubuntu:latest"
)

# Initialize the agent with a model and the computer as a tool
agent = ComputerAgent(
    model="anthropic/claude-3-5-sonnet-20241022",
    tools=[computer]
)

# Give the agent a simple task
messages = [{"role": "user", "content": "Open the terminal and list the files in the current directory."}]

async for result in agent.run(messages):
    print(result)

Here's a quick breakdown:

  1. We import the ComputerAgent and Computer classes.
  2. We initialize a Computer, pointing it to a local Docker container.
  3. We create a ComputerAgent, equip it with a model, and give it the computer as a tool it can use.
  4. We define a task and use agent.run() to execute it.

For a deeper dive into the SDK and its capabilities, check out the developer documentation in our Quickstart | Cua Docs.

Step 4: Run Your Agent and Watch it Work

Save the code as a Python file (e.g., run_agent.py) and execute it from your terminal.

You will see the agent process your request, perform the actions in the sandboxed environment, and stream its results directly to your console.

Congratulations! You have successfully built and run your first computer use agent with Cua.

Advanced Cua Features for Powerful Automations

Once you've mastered the basics, Cua offers powerful features to build more complex and efficient workflows for real-world scenarios.

App-Use: Focus Your Agent on Specific Applications

App-Use is an experimental feature that allows you to create lightweight virtual desktops limited to specific applications. This has several key benefits:

  • Improved Precision: The agent's focus is constrained to a single application, reducing errors.
  • Parallel Workflows: You can run multiple specialized agents in parallel—for example, one agent researching in a browser while another drafts an email.
  • Lightweight Performance: It avoids the overhead of spinning up entirely new containers for each task.

This feature is perfect for workflows that require an agent's undivided attention on one application at a time. To learn more, read our detailed post on App-Use: Control Individual Applications with Cua Agents.

Automating Your iPhone

For a truly advanced use case, you can control an iPhone through the iPhone Mirroring app on macOS. This allows you to automate tasks on a mobile device, such as sending messages, managing settings, or testing iOS applications.

Important Note: This method interacts with a physical device. We strongly recommend running the agent within a virtual machine for safety, as an untrained agent could perform unintended actions on your phone.

Conclusion: The Future is Automated with Cua

Cua makes building powerful computer use agents accessible, fast, and, most importantly, safe. By combining sandboxed environments with a simple yet powerful SDK, we provide a robust platform for developers to explore the next generation of AI automation. Its open-source nature and vibrant community make Cua a leading choice for anyone looking to build a computer use agent on GitHub.

Ready to start building? Dive into our documentation, join the community, and see what you can create. The future of how we interact with technology is being built today, and with Cua, you can be a part of it.

Explore our Home | Cua Docs to continue your learning journey.

Citations

[1] https://github.com/trycua/cua

[2] https://github.com/trycua/cua/blob/main/README.md