Plain-AyI
← All projects
AI agents

ESP32 iPhone Agent

Lets an AI vision model operate a real iPhone using a small, inexpensive ESP32 circuit board plugged into the phone.

What it does

This project lets an AI model use an iPhone the way a person would. A small ESP32-S3 board (a tiny, low-cost computer chip) plugs into the phone. The board receives the phone's screen through Screen Mirroring and acts as a USB mouse and keyboard.

An app on the same iPhone sends screenshots to an AI model that can understand images. The model decides what to do next, such as tap, swipe, type or go home, and the board carries out those actions. You type a goal like "Open Settings and go to Wi-Fi" and press Start.

Every run is saved on the phone so you can replay it step by step. The makers note limits: the mirrored screen is small and slow, some screens like passcode entry can't be seen, typing is English letters only, and sideways (landscape) apps aren't supported yet.

Who it's for

Hobbyists and tinkerers who want to experiment with AI that controls a phone, and who are comfortable flashing a circuit board and building an iPhone app.

Can I use it without coding?

Developers only. You need a Mac with Xcode to flash the board and build the iPhone app yourself using command-line tools. New to this? Start here.

Can I run it?

Cost
Free, but needs an AI accountThe software is free, but it uses an online AI service (like OpenAI or Anthropic), which usually charges for use.
Runs on
Phone, Mac
Privacy
Your choice: run it on your device or use an online service
API key
Needed. An API key is a password-like code from an online AI service. What's that?

Try it

  1. Get an ESP32-S3 board with PSRAM (the makers tested a Seeed Studio XIAO ESP32S3) and flash it from a Mac using the project's flash script.
  2. Build and install the ESP32 Agent app on your iPhone with Xcode.
  3. Plug the board into the iPhone, open the app, and follow its four setup steps, including entering your AI model address and key.

Using Claude Code, Cursor or another AI coding assistant? Paste the prompt and it will walk you through the setup.New to this?

Before you install

  • Only download from the official GitHub page. Copies with similar names exist for many popular projects.
  • Plain-AyI is an independent guide and doesn't host or review the software it lists. Please check each project's official page before installing.
  • License (MIT): Very permissive: free to use, change and share, including for business.
  • Keep your API key private. Never paste it into websites or share it, and set a spending limit with the AI service.
  • More on staying safe