Abstract:

This document describes the Gemini 2.5 Computer Use model, a tool designed for building browser control agents that can automate tasks by interacting with graphical user interfaces (GUIs). The model operates by analyzing screenshots to "see" the screen and generating specific UI actions, such as mouse clicks and keyboard inputs, which are then executed by a client-side application. The core of its implementation is an "agent loop" where the model receives the current screen state, suggests an action, waits for the client to execute it, and then receives a new screenshot to determine the next step. The documentation details the implementation process, including the required secure execution environment, the list of supported UI actions (e.g., click_at, type_text_at), and the ability to add custom functions for non-browser environments. A significant emphasis is placed on safety and security, detailing a built-in safety system that may require human confirmation for potentially risky actions, and outlining best practices for developers, such as using sandboxed environments and implementing human-in-the-loop (HITL) workflows.

Building Automated Agents with Gemini Computer Use

• What is Gemini Computer Use?

  • It is a preview model (gemini-2.5-computer-use-preview-10-2025) and tool that allows developers to build agents that can see a computer screen via screenshots and act upon it using generated UI actions like clicks and keyboard inputs.
  • Key Use Cases: Automating data entry, testing web applications, and conducting automated research across websites.

• The Core Agent Loop:

  • The system works in a continuous four-step loop to complete tasks.
  • 1. Send Request: The developer sends a prompt (the user's goal) and a screenshot of the current GUI to the model.
  • 2. Receive Response: The model analyzes the input and suggests a UI action in the form of a function_call (e.g., "click at coordinate (x,y)").
  • 3. Execute Action: The developer's client-side code (e.g., using a library like Playwright) receives and executes the suggested action in the target environment.
  • 4. Capture New State: After the action is performed, a new screenshot is captured and sent back to the model, and the loop repeats until the task is complete.

• Implementation and Functionality:

  • Prerequisites: Developers must set up a secure, sandboxed execution environment (like a container or VM) and implement client-side logic to handle the actions generated by the model.
  • Supported Actions: The tool has a wide range of built-in UI actions, including open_web_browser, navigate, click_at, type_text_at, scroll_document, and drag_and_drop.
  • Customization: Developers can extend the model's capabilities by adding their own custom functions (e.g., open_app for a mobile environment) or excluding predefined actions they don't want the agent to use.
  • Coordinate System: The model outputs normalized coordinates (on a 0-999 scale) which must be converted to actual pixel values by the client application. The recommended screen size is 1440x900.

• Safety and Security:

  • Safety Decisions: The API includes a safety system that may classify a proposed action as require_confirmation. This happens for potentially risky actions like interacting with a CAPTCHA.
  • Human Confirmation Required: When a require_confirmation decision is received, the application must prompt the end-user for confirmation before executing the action. Bypassing this is against the terms of service.
  • Best Practices: Developers are strongly encouraged to follow safety best practices, including using a human-in-the-loop (HITL) approach, running agents in a secure sandboxed environment, sanitizing inputs, and using allowlists/blocklists for websites.