Computer use and remote screen
Codync can do two things with your computer's screen:
- Remote screen: you see the screen on your iPhone, or on another computer's desktop app, and control it with touch, the mouse and the keyboard.
- Use the computer: a bot opens apps, reads windows, clicks and types, the way you would.
Both run through Codync Screen, a small helper on the computer that the desktop app sets up. You can watch a bot work from your phone and take over at any time.
Set up computer access
Permissions are granted on the computer itself. In the desktop app, open Settings → Computer access. It's also part of the first setup (you can choose Set up later for each permission), and in the menu bar or tray menu under Remote screen → Set up computer access….
During first setup, use the capsule above the content to switch between steps. On macOS, it contains Codync Screen, Screen recording, Control apps, and GitHub Star. Linux combines screen sharing and control; Windows shows computer control availability. The final GitHub Star invitation is optional: choose Get started to finish without starring. Permissions can be completed later in Settings.
- macOS: press Enable Codync Screen. If macOS asks, allow Codync to run in the background in Login Items. Then allow Screen recording and Control apps. In System Settings, look for Codync Screen; depending on your macOS version, control is listed under Accessibility or Device Control and Data Access. Come back to Codync after granting, and let the helper restart if asked.
- Linux: press Choose screen and allow control. The system's screen-sharing dialog asks you to pick a screen and allow mouse and keyboard access. If Codync Screen restarts, open Computer access again to share the screen once more. A plain X display without that dialog (a server's Xvfb, a Codync Cloud computer) needs nothing: it's ready as soon as Codync Screen runs.
- Windows: bots can use apps without extra permissions. Viewing the screen from your phone isn't available on Windows yet.
The iPhone app and the terminal UI show whether access is set up and point you to Computer access on the computer; they can't grant it.
Turn remote screen on or off
Remote screen is on by default on macOS, and on Linux when a graphical session (Wayland or X11) is running. On a headless Linux server it stays off.
To change it, use the menu bar or tray menu on the computer: Remote screen → Enable remote screen. Only the computer itself can switch it on; a phone can't. Codync remembers your choice.
View and control your screen from the iPhone
Open the screen in either of two ways:
- In a chat, tap the Computer button at the top right.
- Tap the connection status at the top of the Bots tab (such as 1 connected), choose Manage computers, and tap Screen on the computer.
You have control right away, unless that chat's bot is using the computer at that moment (then you watch first, as described under "Watch a bot work" below). While you have control, bots can still look at the screen, but they can't act until you hand it back.
The buttons along the top:
| Button | What it does |
|---|---|
| Close | Ends the session and hands control back to the bots |
| Rotate | Turns the view, even with rotation lock on (it otherwise follows how you hold the phone) |
| Keyboard | Shows the keyboard, with a row of modifier keys |
| Touch mode | Switches between trackpad (drag to move the pointer) and direct (tap where you want to click) |
| Clipboard | Send my clipboard to the computer, or Copy from computer |
| Fit screen | Appears after you zoom in; zooms back out |
| Displays | Appears with more than one display; picks which one you see |
| Take over / Hand back to bots | Takes control from the bots, or gives it back |
If the computer allows viewing but not control, the screen says "View only" and points you to Computer access on the computer.
View and control another computer from the desktop app
The desktop app shows the screen of your other computers too: a cloud computer, or one you reach over SSH. Choose Screen on that computer's heading in the sidebar, or on its card in Settings → Computers. (Your own computer's screen is the one in front of you, so it has no Screen button.)
You have control as soon as it opens, and bots can still look. The mouse and keyboard work as they do on that computer: clicks, double-clicks, drags, scrolling, shortcuts, and typing in any language, input methods included. On a Mac, ⌘ acts as Ctrl on a Linux or Windows computer, so ⌘C and ⌘V copy and paste there. Pasting sends your clipboard first; when that computer copies text, Copy from computer appears at the top. Hand back to bots gives control back without closing (your choice holds when the screen reconnects), Displays picks a display when there's more than one, and closing the sheet ends the session.
How the screen travels
- Video and input go over WebRTC and stay encrypted between your phone or desktop app and Codync Screen. The setup messages travel over the same end-to-end encrypted connection as your chats.
- On a direct connection (same Wi-Fi, or Tailscale), Codync uses no STUN or TURN server. On a Mac, the picture is encoded at up to 16 Mbps and 60 fps.
- Through the Codync relay, the stream may be forwarded by Cloudflare's TURN service, which passes along encrypted packets and never sees your screen's content. This needs the computer linked to a signed-in account; otherwise the phone asks you to sign in on the computer. The desktop app goes through the relay only for a computer in your account (a cloud computer, for one); a computer it reaches over SSH always goes directly. Relayed sessions are encoded at up to 4 Mbps and 30 fps, even if a direct path turns up later.
- The screen follows the computer's Connection choice in Manage computers: Auto tries a direct path first, then Cloudflare's TURN service; Cloudflare first does the reverse; Wi-Fi or Tailscale only never asks for TURN. A computer in no account has no choice: the screen goes only directly.
- With updated iPhone and computer builds, an established screen connection can continue through a temporary chat reconnection. Setup failures reconnect with a fresh screen session.
- Revoking the phone's access or turning off remote screen stops the picture and input.
A bot's computer
Each bot runs on one computer. The Computer tab in its details (click or tap the bot's name at the top of the chat) shows that computer: its name, whether it's connected, and a live picture of its screen, so you can watch the bot work without opening anything else. Click or tap the picture to open the full screen: you watch first if that bot is using the computer, and choose Take over to control it. A cloud computer that's asleep wakes when you look, and the picture shows "Waking the computer…" until it's ready.
Nothing streams when you open the tab: click or tap Click to see the screen (Tap to see the screen on a phone)
to start the picture. It streams only while the tab is open, and stops while the full screen is open. When nobody
touches Codync for 5 minutes, the picture and the full screen pause and show "Paused. Click to continue." ("Paused. Tap to continue." on a phone); click or tap
to pick up where you were. On the desktop app, the
computer the app runs on has no picture: its screen is the one in front of you. The terminal UI's Computer page (press O
on the bot) shows the computer and Use the computer, without a picture.
Let a bot use the computer
Turn on Use the computer for the bot. It's off by default.
- Desktop app, iPhone and Android: open the bot's details (click or tap the bot's name at the top of the chat) and find Use the computer on the Computer tab.
- Terminal UI: press
eon the bot and toggle Use computer. The form also shows whether computer access is ready.
On macOS and Linux, remote screen must be on for bots to use the computer. On Windows it's always available.
What a bot can do
With Use the computer on, the bot gets tools to:
- list apps and windows, read what's in a window, and zoom into part of the screen
- open apps
- click, double-click, right-click, drag and scroll
- type text, press keys and shortcuts, set values in fields, and pick menu items
- type one of your saved sign-ins (see below)
On macOS, Windows, and Linux with X11 or Sway, the bot works on apps in the background: your pointer and the app in front stay yours. On macOS, windows on another Space can't be worked on in the background; the bot is told so and can act in the foreground instead.
Bots don't get tools to bring an app to the front, quit apps, read your clipboard, or drive a browser through its automation interface.
The rules
- One bot at a time. Another bot that tries is told the computer is in use.
- You come first. While you've taken over from your phone, bots can only look until you hand back control.
- Approvals still apply. The bot's Permissions setting and your agent's own approval rules cover each action.
Watch a bot work
While a bot uses the computer, the Computer button in its chat pulses. Tap it to watch: the screen shows "Bot is using the computer". Tap Take over to take control; from then on the bot can only look. Tap Hand back to bots when you're done.
Platform notes
- macOS: bots use Codync Screen's Screen recording and control permissions, so macOS asks for nothing new.
- Windows: windows of apps running as administrator are out of reach. Typing saved sign-ins isn't available on Windows yet.
- Linux with X11 or Sway: the computer-use tools need libX11, libXi and libxkbcommon.
- Linux with GNOME, KDE or another Wayland desktop: bots get a simpler set of tools that take screenshots and click and type on screen positions, using the screen you shared in Computer access.
- Linux remote screen: Codync Screen needs glibc 2.39 or later and GStreamer 1.22 or later with the base, good and bad plugins; a desktop with a screen-sharing dialog (GNOME, KDE) also needs PipeWire and xdg-desktop-portal, while a plain X display needs neither. The install script, Homebrew and
codync-host updateput it next to the host.
Saved sign-ins
When a bot needs to sign in to a website or app, it can ask you with a card in the chat. Enter Username or email and Password or op:// reference, then Save. You can also add them under Extensions → Credentials → Sign-ins.
The bot then types the sign-in into the page itself and never sees the password. Codync types only when the focused page belongs to that site (or one of its subdomains), or the front app has that name, and it types a password only into a password field. Sign-ins are encrypted on the computer like connector credentials.
Password prompts in a terminal (such as sudo or ssh) aren't covered.