Shared NPU scheduling for tracking and VLM inference

Two jobs, one accelerator

A hub in a home runs a continuous, latency-critical task—keeping a camera on a moving animal—and an occasional, expensive one: reading a scene with a vision-language model in order to say something about it. On a device with a single small accelerator, these cannot both be resident whenever they feel like it.

Left unmanaged, the interaction is not a graceful slowdown. Tracking stalls at the moment the room becomes interesting, or the longer inference is starved and never produces a usable reading, or both proceed at half strength and the camera develops a wandering, unreliable gait. The household version of the question is simple: while the hub is thinking, does the camera still follow, and afterwards does it come back.

Why a timer is not a handoff

The convenient scheduling primitive is a deadline: give the interpreter a slice, and assume it has finished when the slice expires. On real inference stacks this assumption is wrong in an unsafe direction. A request that has been asked to stop is not the same as a request that has released the accelerator, and resuming on the strength of a timer means two workloads briefly believe they own the same silicon.

We therefore treat the accelerator as having exactly one owner at a time, with transitions driven by observed state rather than elapsed time. Tracking holds the device while there is an animal to follow. A longer look may begin only once the scene is quiet enough to justify it and the camera has been commanded to hold position. Tracking resumes only after the interpreter is confirmed paused and the device explicitly released—not when a lease looks stale, and not when a client connection drops, since a lost connection is evidence of a network problem rather than permission to proceed.

A longer look has internal phases

Hardware-in-the-loop measurement on the development board showed that the interesting variable is not how fast a stop request is noticed but how long the underlying stage takes to become quiet. Revocation was detected within tens of milliseconds in every stage we exercised—between roughly 60 and 85 ms.

Reaching a genuinely paused state varied by more than an order of magnitude depending on what the model was doing. During token generation the request could be suspended in about 150 ms. During image encoding it took a little over a second, and during the initial prompt-processing stage it took several seconds. Throughout those intervals the tracker stayed paused rather than being handed a device that was still busy, and generation then continued as the same request afterwards rather than being restarted from the beginning.

We report the fast detection figure and the slow quiescence figure together on purpose. Publishing only the first would imply an accelerator that frees itself in under a tenth of a second, which is not what the hardware does.

Yielding to the household

Preemption is driven by things a person would recognise. Opening a live view is a demand for the camera and revokes the longer look. Fresh activity reported by the collar does the same, on the argument that a moving animal is worth following now and interpreting later. A scene that changes materially while the hub believed the room was empty also invalidates the premise on which the longer look was permitted.

After release, tracking recovered to roughly 17 fps of steady following, with a median inference around 51 ms, and repeated sampling of the runtime after resumption showed the tracker consistently active rather than intermittently reclaiming the device.

A quiet room has to actually be quiet

Because a longer look is only allowed to start when the scene is empty enough, the emptiness test is part of the safety argument. Our first implementation sampled one pixel per block, which on a static empty room produced enough apparent change to trigger a false scene event on the very next frame.

Averaging a small set of samples per block and judging the fraction of changed blocks removed the sensitivity: through a real inference lasting over a thousand frames, the detector reported no scene change, and no false trigger recurred across the remaining phase and live-demand runs. A cheap detector that fires spuriously is worse than no detector, because it makes preemption look flaky and invites someone to widen the threshold until real changes are missed too.

Mechanism is not comprehension

The same evidence run that validated the scheduling contract also declined to validate the model behind it. Asked to produce a structured reading of a real stored frame, the model returned malformed output on three consecutive attempts, and nothing was written to the record. That result is reported as a failure of the interpretation path, not folded into the resource work that passed.

Keeping the two separate is the point of the note. A clean handoff is a mechanical guarantee: the camera can return to a living animal. Whether the hub understood what it saw is a claim about language and evidence, and it has to be earned on its own terms.