An accessibility tree AI agents can drive: UI testing by label

Every control in an Osprey UI machine has a role and a label. This video shows the accessibility tree that produces, and end-to-end UI tests that tap controls by label with real pointer events.

An accessibility tree AI agents can drive: UI testing by label (2:08). Captions are on; choose a chapter to jump to it.

The full write-up, with the code and its limits, is An accessibility tree AI agents can drive.

Osprey is actively evolving.

Transcript

  1. There are two ways for software to use an interface it did not write.
  2. It can look at pixels: take a screenshot, guess where the Play button is, and click there.
  3. Or it can read the accessibility tree: every control's role and label. The button labelled Play survives a redesign that moves it.
  4. In Osprey, an interface written as a UI machine produces its own tree. Every hit area on this player names its role and its label.
  5. Semantics turns the live controls into the tree: role, filled-in label, bounds, and a slider's value.
  6. Act operates a control by label, the way a screen reader or an agent would. An unknown label changes nothing.
  7. A test taps by label, and a tap is a real pointer press and release at the control's bounds.
  8. If a hit area is misplaced or covered, the test fails, just as a user's click would.
  9. Read it as a user would. Before a file is chosen, Play sends nothing to the TV.
  10. Open a file, press Play, and the player connects and launches.
  11. Tap halfway along the volume slider, and the volume is a half.
  12. Tap a quarter along a two hundred second track, and it shows fifty seconds.
  13. The test drives the app's real start and dispatch, with no window, and a handler plays the TV. The second journey switches skins: the pixels change, the labels must not.
  14. All four journeys pass, the same way every run.
  15. The GPUI desktop host maps the same tree to AccessKit, and turns a screen reader's click into the same pointer events.
  16. So the app cannot tell a screen reader or an agent from a hand.
  17. An agent needs no vision. It reads a short list of what can be done right now, picks one by name, and acts. And the labels it uses are tested on every build.
  18. What is not done yet. There is no ready-made MCP server for agents. The tree comes from UI machines only, and only the GPUI host maps it today.
  19. Build Osprey, and run the casting journeys. Osprey is actively evolving. Try it on your projects.