Open Record/Replay

Teach Computer Use agents by demonstration.

Record a macOS workflow once, save the evidence, and let an agent turn it into a reusable skill.

English | 简体中文

Open Record/Replay is a local-first macOS recorder for workflows that are easier to show than to write down. It captures a user's real desktop actions into structured artifacts such as session.json and events.jsonl, validates the recording quality, and packages the evidence for an agent's skill creation flow.

The goal is simple: if a user can demonstrate a desktop workflow once, an agent should have enough evidence to learn it.

flowchart LR
  A["User demonstrates a macOS workflow"] --> B["Open Record/Replay records events"]
  B --> C["session.json + events.jsonl"]
  C --> D["Validate recording quality"]
  D --> E["Prepare skill input package"]
  E --> F["Agent creates its own reusable Computer Use skill"]

Why This Exists

Some workflows are hard to describe as prompts:

• They depend on visible desktop UI.
• They involve file pickers, menus, modals, or drag-and-drop.
• They span multiple apps.
• They depend on personal workspace layout or team-specific conventions.
• They are easier to learn from a demonstration than from a written runbook.

Open Record/Replay gives agents a concrete evidence stream instead of asking them to infer the workflow from a vague description.

What You Can Record

Open Record/Replay can capture action-level evidence for workflows such as:

• Sending files or images through a desktop chat app.
• Creating a document and sharing its link.
• Opening a browser page, searching, and starting the right media.
• Moving between browser and desktop apps.
• Repeating a UI process that is not covered by a clean API or connector.

The recorder can capture:

• window.changed
• mouse.click
• mouse.drag
• keyboard.text_input
• keyboard.submit
• selection.changed
• app/window attribution
• UI targets
• selected items
• Accessibility tree or diff context

events.jsonl is the primary evidence. Screenshots are not part of the current core recording path.

Quick Demo

This is the typical flow for teaching an agent a new desktop workflow.

git clone https://github.com/humblebanana/open-record-replay.git
cd open-record-replay
npm install
npm run check

Check macOS permissions:

node bin/orr.js permissions check

Request missing permissions:

node bin/orr.js permissions request

Start a recording:

node bin/orr.js record start --name send-file-demo --out runs --request-permissions

Now demonstrate the workflow on your Mac. For example:

1. Open a desktop chat app.
2. Choose a recipient or channel.
3. Attach a local file.
4. Confirm the upload.
5. Send a short follow-up message.

When finished, stop the recording:

node bin/orr.js record stop latest

Validate the evidence:

node bin/orr.js session validate-recording latest

Prepare a skill input package:

node bin/orr.js skill prepare latest --runs runs --out skill-inputs

The package will be written to:

skill-inputs/<session-id>/
├── README.md
├── events.jsonl
└── session.json

Give this directory to the current agent's skill creation flow.

Output Artifacts

Recording output:

runs/sessions/<session-id>/
├── session.json
├── events.jsonl
├── orr_session.json
└── recording_manifest.json

Skill input package:

skill-inputs/<session-id>/
├── README.md
├── events.jsonl
└── session.json

session.json records the recording boundary, timing, and event path. events.jsonl is the source of truth for what happened during the demonstration.

Agent Integration

This repository includes a host instruction skill:

skills/open-record-replay/SKILL.md

An agent should use it to understand when to start recording, when to stop, how to inspect events.jsonl, and how to hand the evidence package to its own skill creation flow.

Expected agent flow:

1. Check or request required macOS permissions.
2. Start recording only when the user is ready.
3. After recording starts, stop the current turn and wait for the user to finish the demonstration.
4. Stop the recorder when the user says the demonstration is complete.
5. Read session.json and events.jsonl.
6. Validate the recording.
7. Prepare the skill input package.
8. Use the agent's own skill creation system to create and validate the final skill.

CLI Reference

node bin/orr.js permissions check
node bin/orr.js permissions request
node bin/orr.js record start --name my-workflow --out runs --request-permissions
node bin/orr.js record stop latest
node bin/orr.js session list
node bin/orr.js session inspect latest
node bin/orr.js session events latest
node bin/orr.js session validate-recording latest
node bin/orr.js skill prepare latest --runs runs --out skill-inputs

The stable public path is recording, validation, and skill input packaging.

Requirements

• macOS.
• Node.js 18+.
• Swift toolchain / Xcode Command Line Tools.
• Accessibility permission.
• Input Monitoring permission.

The core recorder does not require Screen Recording.

Privacy

Recordings are local by default, but events.jsonl can contain sensitive data:

• Window titles.
• URLs.
• Typed text.
• Selected text.
• File names.
• Local paths.
• Accessibility tree text from apps and web pages.

Review recordings before sharing them. Do not publish raw recordings that contain secrets, private documents, customer data, internal URLs, or personal information.

See Privacy.

Status

Open Record/Replay is alpha software.

Current public scope:

macOS native recorder
+ CLI
+ session.json / events.jsonl
+ recording validation
+ skill input package
+ host-agent skill creation handoff

Future work may include richer adapters or optional visual evidence. They are not part of the current stable public path.

Documentation

• Installation
• Agent Usage
• Recording Data Contract
• Privacy
• Release Checklist
• Contributing

---

Simple Chinese

English | 简体中文

Open Record/Replay lets a Computer Use Agent learn a macOS desktop workflow through a single real demonstration.

It records a user's real actions on their Mac into session.json and events.jsonl, checks recording quality, and packages them into a Skill input package that the Agent can read. The final Skill is not generated directly by Open Record/Replay; it is produced by the current Agent using its own Skill creation flow.

The core goal is simple: if a user can demonstrate a workflow once, the Agent should have enough evidence to learn it.

flowchart LR
  A["用户演示 macOS 工作流"] --> B["Open Record/Replay 录制事件"]
  B --> C["session.json + events.jsonl"]
  C --> D["验证录制质量"]
  D --> E["打包 Skill 输入"]
  E --> F["Agent 创建自己的可复用 Computer Use Skill"]

Why It's Needed

Some workflows are hard to write as prompts:

• It depends on the real desktop UI.
• It involves file pickers, menus, modals, or drag-and-drop.
• It spans multiple apps.
• It depends on a personal workspace layout or team-specific conventions.
• A single demonstration by the user is clearer than writing a long runbook.

What Open Record/Replay gives the Agent is not a vague description, but a real stream of event evidence.

What Can Be Recorded

Typical scenarios:

• Sending files or images through a desktop chat app.
• Creating a document and sharing its link.
• Opening a web page, searching for content, and playing the specified media.
• Switching operations between a browser and desktop apps.
• Reproducing a UI flow that has no stable API or connector.

Can capture:

• window.changed
• mouse.click
• mouse.drag
• keyboard.text_input
• keyboard.submit
• selection.changed
• App / window attribution
• UI target
• Selected files or text
• Accessibility tree / diff context

events.jsonl is the most critical evidence. Screenshots are not part of the current core recording path.

Quick Demo

Install and check:

git clone https://github.com/humblebanana/open-record-replay.git
cd open-record-replay
npm install
npm run check

Check macOS permissions:

node bin/orr.js permissions check

Request permissions:

node bin/orr.js permissions request

Start:

node bin/orr.js record start --name send-file-demo --out runs --request-permissions

Then demonstrate your workflow on your Mac. For example:

1. Open a desktop chat app.
2. Choose a contact or group chat.
3. Attach a local file.
4. Confirm the upload.
5. Send a follow-up message.

Stop recording:

node bin/orr.js record stop latest

Validate:

node bin/orr.js session validate-recording latest

Prepare the Skill input package:

node bin/orr.js skill prepare latest --runs runs --out skill-inputs

Output goes to:

skill-inputs/<session-id>/
├── README.md
├── events.jsonl
└── session.json

Hand this directory to the current Agent's Skill creation flow.

Output Artifacts

Output:

runs/sessions/<session-id>/
├── session.json
├── events.jsonl
├── orr_session.json
└── recording_manifest.json

Skill input package:

skill-inputs/<session-id>/
├── README.md
├── events.jsonl
└── session.json

session.json records the recording boundary, timing, and event path. events.jsonl is the source of truth for what the user actually did.

How the Agent Integrates

The repository includes an instruction Skill for Agents:

skills/open-record-replay/SKILL.md

The Agent should use it to understand when to start recording, when to stop, how to read events.jsonl, and how to hand the evidence package to its own Skill creation flow.

Flow:

1. Check or request the necessary macOS permissions.
2. Start recording only when the user is ready.
3. Once recording starts, the Agent stops its current turn and waits for the user to finish the demonstration.
4. When the user says it is complete, the Agent stops the recording.
5. Read session.json and events.jsonl.
6. Validate the recording quality.
7. Prepare the Skill input package.
8. Use the current Agent's own Skill creation system to generate and validate the final Skill.

CLI Commands

node bin/orr.js permissions check
node bin/orr.js permissions request
node bin/orr.js record start --name my-workflow --out runs --request-permissions
node bin/orr.js record stop latest
node bin/orr.js session list
node bin/orr.js session inspect latest
node bin/orr.js session events latest
node bin/orr.js session validate-recording latest
node bin/orr.js skill prepare latest --runs runs --out skill-inputs

The current stable public path is: record, validate, package the Skill input.

Requirements

• macOS.
• Node.js 18+.
• Swift toolchain / Xcode Command Line Tools.
• Accessibility permission.
• Input Monitoring permission.

The core recorder does not require the Screen Recording permission.

Privacy

Recordings are stored locally by default, but events.jsonl may contain sensitive data:

• Window titles.
• URLs.
• Typed text.
• Selected text.
• File names.
• Local paths.
• Accessibility tree text from apps and web pages.

Review before sharing. Do not publish raw recordings that contain secrets, private documents, customer data, internal URLs, or personal information.

See Privacy.

Project Status

Open Record/Replay is currently an alpha release.

Current public scope:

macOS 原生录制器
+ CLI
+ session.json / events.jsonl
+ 录制质量验证
+ Skill 输入包
+ 交给宿主 Agent 创建最终 Skill

Future work may include richer adapters or optional visual evidence. Not part of the current stable path.

Docs

• Installation
• Agent Usage
• Recording Data Contract
• Privacy
• Release Checklist
• Contributing