Introduction to Copilot Computer Use

GitHub has officially announced that computer use is now available in public preview for the GitHub Copilot CLI and the GitHub Copilot app on macOS and Windows. This new capability allows Copilot to interact directly with desktop applications on behalf of the user, significantly expanding the scope of tasks that can be automated. Further details are available from GitHub Changelog in the original source material.

According to the official GitHub Changelog, the integration enables the assistant to handle legacy and graphical user interface (GUI) software that traditionally lacks an API, command-line interface, or Model Context Protocol (MCP) integration.

Capabilities and Application Workflows

With the newly introduced tools, Copilot can perform a wide variety of hands-on actions across desktop environments. These actions include reading accessible app content and visual context, clicking controls, entering and editing text, pressing keys, scrolling, dragging, and navigating complex workflows across different software applications.

For instance, the system can utilize computer-use tools to navigate an expense-report workflow inside a browser like Safari. Users achieve the best results when they clearly describe the desired outcome, the specific applications involved, and any critical constraints. Tasks can range from summarizing notifications in a browser to updating presentation content or moving data through a desktop application workflow.

User Controls, Security, and Permissions

GitHub emphasizes that users maintain full control over the assistant's actions. Copilot always asks for explicit approval before taking control of any application. Furthermore, users retain the ability to review or reset applications that have been granted permission to always be allowed.

On macOS systems, the computer use feature guides users through the necessary Accessibility and Screen Recording permissions required to function. In addition, organization-managed settings provide administrators with the option to disable the feature entirely across corporate environments.

How to Enable and Manage Computer Use

Enabling the functionality varies depending on whether you are using the command line interface or the graphical desktop client. In the Copilot CLI, users can simply run the command to activate it, check its status, or turn it off when finished.

For the desktop software, users can navigate to the application settings, select the computer use menu option, and turn on the toggle to enable the feature. Alternatively, the desktop app also supports turning the capability on via quick command input.

Sources

• GitHub ChangelogGitHub Copilot can now interact with desktop apps with computer use