Freeze your screen, mark regions, get pixel-exact coordinates and crops

60sSixty seconds

pixelcoords
# screen freezes; drag shapes, A labels, S saves
# → Downloads/pixelcoords-captures/<timestamp>/
#   session.json  screenshot-0.png  cutout-primary-0.png  cutout-inverse-0.png  crop-0-submit.png

pixelcoords assert --session <dir> --point 812,440 --expect submit
# exit 0: that point is inside the region you labeled "submit"
# --stdin scores a whole trajectory in one process, one point per line

pixelcoords resolve --session <dir> --label submit --units auto
# where to click, right now, in the units your input API actually wants

pixelcoords emit --session <dir> --format pyautogui
# ready-to-paste click code, coordinate conventions already handled

pixelcoords wait --session <dir> --label dialog --for match
# block until that region is back on screen; exit 1 if it never arrives

pixelcoords diff --session <dir> --against baseline/
# did those regions still look right? visual regression, per region

pixelcoords find --session <dir>
# the UI moved? every region re-located by its saved crop, deltas included

pixelcoords mcp
# serve all of the above to an LLM over stdio
# read-only: it asks about sessions you marked, it cannot make one

pixelcoords resume
# pick any saved session, keep editing it

loopThe tool is a loop

{ "schema": 1,
  "selections": [{ "shape": "rect", "label": "submit",
    "px": { "x": 812, "y": 440, "w": 96, "h": 40 } }] }

{ "command": "assert", "ok": true,
  "results": [{ "hit": true, "space": "global" }] }   # exit 0

{ "label": "submit", "scale": 2.0, "units": "logical",
  "point": { "x": 406, "y": 230 } }

pyautogui.click(812, 460)  # submit

{ "ok": true, "polls": 7, "elapsed_ms": 3204,
  "results": [{ "score": 0.997, "matching": true }] }

{ "masked_px": 400, "changed_px": 3,
  "changed_pct": 0.75, "mean_delta": 172.67 }

{ "found": true, "score": 0.998,
  "delta": { "dx": 0, "dy": -120 } }

session.json  screenshot-0.png  crop-0-submit.png  → editable again

mcpDrive it with your LLM

The loop above answers the questions a machine asks. Reaching it still took a human — wire a subprocess call, remember the flag spellings, parse stdout. pixelcoords mcp serves the same commands over the Model Context Protocol on stdio, so a model calls them directly. Point a client at the binary:

{
  "mcpServers": {
    "pixelcoords": { "command": "pixelcoords", "args": ["mcp"] }
  }
}

A miss is an answer, not a failure. The exit codes survive the trip: 0 and 1 both come back as ordinary results, so a miss, a timeout, or an over-tolerance diff arrives with ok: false and no error flag. Only a malformed question is an error. Backwards, every negative answer looks like a broken tool and the model retries instead of reacting.

It is read-only. No tool opens the overlay, renames a session, or edits one — marking regions is something a human does. The shape is mark once, run many. Asking a marked session where to click reads a file and needs no screen-recording permission; the tools that do capture say so in their own descriptions. Full reference.

outputsWhat a save writes

pixelcoords-captures/20260728-182121-117/
├── session.json              # versioned schema: three coordinate spaces, DPI scale
├── screenshot-0.png          # one full frozen capture per monitor
├── cutout-primary-0.png      # the frame with only the selections visible
├── cutout-inverse-0.png      # the exact complement: selections punched out
└── crop-0-submit.png         # one labeled crop per selection

Crops isolate each region; the cutout pair keeps every region in place on transparency — together they reassemble the screenshot. Files pixelcoords didn't write are never touched.

comparisonWhere it stands

cutout-inverseNon-goals

Knowing what a tool is means knowing what it isn't. These are settled:

• OCR— text extraction is a different product
• Live (unfrozen) measurement— the freeze is the thesis
• Annotation— arrows and blur are a screenshot editor's job
• Recording / GIF capture— the product is the frozen instant, not the timeline
• Cloud, accounts, sharing— offline by design, permanently

platformsPlatform status

Hand-verification is per platform and per release, so the rows above are not a blanket claim. On macOS the overlay was driven by hand through 0.5.0 — multi-monitor and mixed-DPI included, two displays at different scales with every coordinate recorded where it was drawn — and 0.5.1's capture path was checked against a real capture. Windows and Linux were verified by hand through 0.4.0. Nothing since 0.5.1 has had a human overlay run on any platform, which covers the mid-gesture Shift fix and the comfort settings — both change how the overlay behaves, and both are backed by automated tests only. The MCP server is not on that list: it adds no overlay code, and it was driven end to end against a real session. Multi-monitor off macOS, and fractional scaling anywhere, remain test-only. This table is kept honest — claims match runs, and where a run has not happened it says so.

What is checked automatically, on macOS, Windows and Linux, every push: the binary driven against a real capture of the runner's screen — shoot, find, resolve, assert, diff, wait, emit, rename, every shape kind, all six MCP tools — plus display-free tests pinning every exit code across every command. That is a great deal more than it was, and it is still not the overlay. Drawing, dragging, undo, the loupe, snapping, the permission prompts and window targeting need a person at a machine.