Device lab

Fleet Runner

A shelf of old phones, turned into a device lab you can send work to.

GitHub Documentation The eval, in full ← All open source

My plant app identified species by sending photos to a cloud API. I wanted to know whether that could run on the phone instead — offline, at no per-call cost — and on which hardware it would still feel instant. Answering that properly needed more than a benchmark script, so I built the lab first.

One queued job installs a build on every attached device, runs a UI suite across them, benchmarks llama.cpp on real silicon, screenshots a site on two real phone screens and diffs it against a baseline, or drains a battery on purpose and plots the curve. A collector holds the queue, the leases, the artifact store and the results; agents on iOS, Android, desktop and in the browser claim work and report back.

77%: top-1 on-device species ID
7 ms: per image, on a phone CPU
28: workloads across three executors
4: agents, one protocol, no shared code

Agents on a shelf, a machine runner and a browser runner all speaking one JSON protocol to a collector that holds the queue, registry, leases, artifacts, results and scheduler, and serves a dashboard; a host executor on a Mac claims host jobs and drives the shelf from outside. — Every agent speaks the same five calls. The host executor exists for the things an app cannot do to itself — installing a build, tapping through a UI test.

The dashboard's Results screen showing llama.cpp throughput measured on a Samsung tablet: 125.0 tokens per second prefill and 47.4 decode. — Real llama.cpp throughput measured on a Dimensity 9400 — not the synthetic figure the same screen also carries.

What it proves

Three things worth reporting

The answer was yes, and it was specific

An Apache-2.0 ResNet18 hit 77% top-1 and 90% top-5 on held-out images, and the int8 build ran in 7 ms per image on the CPU of a Dimensity 9400 — faster than fp32 with the GPU delegate, and loading in 23 ms instead of 428. The shipping configuration needs no GPU delegate at all, which deletes a whole class of failures.

Four implementations keep the protocol honest

Kotlin, Swift and two TypeScript agents implement the same five HTTP calls and share no code. They also share a synthetic benchmark that is identical on every platform token for token, which is what lets a 2019 Android phone, a current iPhone and a laptop land in the same table comparably. A conformance suite drives a running agent through eight clauses, every one of them something that has actually gone wrong.

The most useful thing it does is decline to produce a number

The iOS Simulator's emulated GPU returned an all-zero logits tensor for a model — silently, no error — and only a second device disagreeing caught it. A phone on battery was measured throttling decode roughly 100×, so a job can demand mains power and the runner refuses rather than reporting a number produced under duress. A wrong number recorded as a result is worse than a failure, because a failure gets investigated.

TypeScriptKotlinSwiftllama.cppCore MLLiteRTSQLitePlaywrightOn-device ML

More open source

Actions Runners →