Low-latency camera-to-tracker transfer through shared memory
A camera that follows a living animal is a control loop, and a control loop is only as good as the freshness of what it reads. Transport is therefore part of the perception problem.
Freshness, not throughput
When a tracker steers a physical camera, the value of a frame decays quickly. A picture that arrives after the animal has moved on does not merely contribute less; it actively steers the head toward where the cat used to be. Optimising this path for average throughput therefore misses the point, because a system can move a great many frames per second and still act on stale information.
This is visible to a household as behaviour rather than as latency: a camera that hunts, overshoots, or settles on an empty cushion while the cat is elsewhere. It is also the difference between a device that feels attentive and one that feels a beat behind the room.
The cost of moving a picture
On a small embedded platform, capture and inference live in separate processes for good reasons—isolation, independent restart, clear ownership of the sensor. The naive way to connect them is to serialise each frame and push the bytes through a socket, which spends a meaningful part of every cycle copying data that both sides already have access to.
Our approach keeps the pixels still and moves only a description. Frames are written into a bounded shared region—four slots of a few hundred kilobytes each, under a megabyte and a half in total—and the consumer receives a small note identifying which slot holds the current frame, its sequence, and how to interpret it. The tracker reads the slot and releases it, so no private backlog of old frames accumulates on the consumer side.
Bounding the region is a deliberate choice about failure. A slow consumer cannot grow an unbounded queue; it loses old frames instead, which for a follow-cam is the correct trade, since the newest valid frame is worth more than a complete history of missed ones. A conventional socket path remains available as an explicit fallback, selected deliberately rather than entered automatically, so a transport problem cannot silently become a different system.
Ordering, and what a consumer is allowed to assume
Sharing a region raises a question a socket answers for free: what happens when the producer reuses a slot while the consumer is still reading it. Our answer is to make the contract explicit rather than probabilistic. A consumer holds a payload only until its next read, sequence numbers are strictly increasing, and a consumer that falls behind observes a gap instead of quietly receiving an older picture as if it were current.
That distinction is what makes the transport safe to reason about downstream. A tracker that can detect a gap can decide for itself whether to coast, re-acquire, or report that it has lost the animal. A tracker handed stale pixels with a plausible sequence number cannot, and it will confidently steer a physical camera on the strength of them.
Doing less when nothing is being followed
Continuous extraction is wasted work when no consumer is asking for a track. Frames are pulled on demand, with a coarse background cadence and an immediate extraction when a consumer connects, so the first frame after a request is never withheld while the pipeline warms up.
The reasoning is thermal and budgetary rather than aesthetic. A hub that produces frames nobody reads spends power and silicon that the same household will later want for a longer look at a moment that actually mattered. On a shared device, restraint in one subsystem is what makes capability in another affordable.
What the board showed
On the development hub, a twelve-second window with capture running and no tracker attached saw the pipeline observe several hundred samples while extracting only fifty-three frames, confirming that demand—not the sensor—sets the working rate. With the tracker attached over a comparable fifteen-second window, every published frame was delivered, and the tracker sustained roughly 18 fps with about 50 ms per inference.
The transport also has to survive the interesting moments. Across those states, and through a longer inference that paused tracking entirely, the frame path recorded no disconnects, and the tracker returned to roughly 16 fps of steady following once the accelerator was handed back. The fallback socket path was exercised on the board as a bounded synthetic run rather than by disturbing the production camera; longer stress runs were performed off-board and we do not present them as board evidence.
Bounds of the claim
Removing one copy from the delivery path is not the same as a zero-copy system. The frame still has to be produced and made available before it can be shared, and that step retains a copy today. The defensible claim is narrower: less of each cycle is spent moving bytes between processes, so more of each cycle is available for perception and for steering.
It is also a statement about mechanism, not about tracking quality. A fresher frame gives the tracker a better chance; whether the tracker follows the right animal is a separate question, judged separately.