FAQ for the computer use tool

What is the computer use tool

Computer use is a tool you can add to your Copilot Studio agents designed to automate tasks on websites and applications. Users provide task descriptions in natural language, and the tool automates mouse clicks and keyboard actions accordingly. For instance, you could instruct the tool to open a PDF and use its contents to fill out a web form. The tool visually interprets the screen and executes the necessary clicks and keystrokes, emulating human interactions.

What can the computer use tool do?

The computer use tool processes screen pixel data to understand on-screen content, and it performs tasks using virtual mouse clicks and keyboard inputs. The tool can manage multi-step workflows, handle unexpected scenarios, and adapt dynamically to screen changes, enabling it to perform various digital tasks such as navigating websites and filling forms without needing specialized APIs.

The computer use tool follows an iterative loop based on three steps:

• Perception: It captures screenshots to visually understand the current state of the computer screen.

• Reasoning: It evaluates the current state, previous actions, and screenshots through a chain-of-thought approach, which allows it to adapt its actions effectively.

• Action: It performs actions like clicking, typing, or scrolling until the task is complete or requires further user input.

What are the computer use tool’s intended uses?

The computer use tool is intended to automate user interactions with websites and applications through a virtual mouse and keyboard, especially useful in scenarios where APIs aren't available.

How was the computer use tool evaluated? What metrics are used to measure performance?

The computer use tool was rigorously evaluated using real-world scenarios throughout its design, development, and deployment phases. Evaluation methods included research, impact studies, and testing across different scenarios to measure accuracy, user trust, and usefulness. Both qualitative and quantitative metrics are regularly monitored to maintain high performance and a positive user experience. The tool’s availability is also ensured by adhering to established SLAs.

What are the limitations of the computer use tool? How can users minimize the impact of the computer use tool’s limitations when using the system?

Users should be aware of the following limitations and usage guidelines when using the computer use tool:

Success rate varies by task

The tool performs best on web-based tasks, with success rates above 80%. However, for desktop application tasks, success drops significantly (around 35%). Reliability decreases with complex interactions or unfamiliar interfaces.

Challenges with complex tasks

While effective for simple actions, the tool might struggle with detailed graphical interfaces or tasks involving intricate text manipulation.

Not for sensitive or high-risk use cases

Computer use isn't designed for the following scenarios:

• Recommending or scoring in sensitive domains (for example, hiring, healthcare, finance)

• Making financial transactions on behalf of users

• Automating harmful actions (for example, spamming, misinformation, DDoS attacks)

• Sharing data beyond organizational boundaries without authorization

System errors or failures are possible, so the tool shouldn't be used in high-risk situations. Misuse – including participation in illegal or unethical activities – should be strictly avoided.

How to minimize limitations

To improve reliability, users should provide clear, detailed instructions when configuring tasks. This helps reduce errors and increases task success rates.

What operational factors and settings allow for effective and responsible use of the computer use tool?

The computer use tool performs best when clear and detailed user instructions are provided. Users should give precise hints about website or application interactions to enhance reliability and accuracy during task execution.

How do I provide feedback on the computer use tool?

Have feedback on the computer use tool? Email computeruse-feedback@microsoft.com.