Plain-AyI
← All projects
AI agents

Self-Operating Computer Framework

Lets an AI model look at your screen and use the mouse and keyboard to finish a task.

Research-backed

What it does

Self-Operating Computer lets an AI model control a computer the way a person does. The model looks at the screen and decides which mouse clicks and key presses will reach a goal you give it.

It works with several multimodal models (AI that can understand both pictures and text), including OpenAI's GPT-4o, Google's Gemini, Anthropic's Claude, Qwen-VL and LLaVA. LLaVA can run on your own computer through a tool called Ollama, though the makers warn its error rate is very high.

Extra modes include voice input for describing your goal, and text-recognition (OCR) that helps the model find buttons by the words on them. The makers say the OCR mode worked best in their tests, so it is the default.

Who it's for

Curious tinkerers and developers who want to try an AI that operates a computer on its own. You should be comfortable using the Terminal.

Can I use it without coding?

With some setup. You install it with one command, run it from the Terminal and paste in an API key; voice mode needs a few more setup steps. New to this? Start here.

Can I run it?

Cost
Free, but needs an AI accountThe software is free, but it uses an online AI service (like OpenAI or Anthropic), which usually charges for use.
Runs on
Mac, Windows, Linux
Computer
An ordinary computer is fine
Privacy
Your choice: run it on your device or use an online service
API key
Needed. An API key is a password-like code from an online AI service. What's that?

Try it

  1. Install the project with Python's package installer, then start it from the Terminal.
  2. On a Mac, allow the Terminal app 'Screen Recording' and 'Accessibility' permissions in System Preferences.

Using Claude Code, Cursor or another AI coding assistant? Paste the prompt and it will walk you through the setup.New to this?

Before you install

  • Only download from the official GitHub page. Copies with similar names exist for many popular projects.
  • Plain-AyI is an independent guide and doesn't host or review the software it lists. Please check each project's official page before installing.
  • License (MIT): Very permissive: free to use, change and share, including for business.
  • Keep your API key private. Never paste it into websites or share it, and set a spending limit with the AI service.
  • More on staying safe

Where people are talking about it