Multimodal learning across visual and wearable signals

The unit of study is a moment, not a file

A household asks a plain question: what was the cat doing just now. Answering it draws on three unlike sources. A camera describes the room from a fixed vantage point. A microphone at the neck hears what the body is close to. An inertial sensor feels how that body moved. Each is partial, and each is partial in a different direction, which is exactly why the combination is interesting.

The failure mode is subtle. If three recordings are treated as one sample merely because they sit in the same folder and carry similar names, a model can be trained on a stretch that was never accompanied by that chewing sound, and on a chewing sound that belonged to a different minute. Nothing crashes. The dataset simply teaches a relationship that did not occur, and the error surfaces much later as a daily summary that mixes eating with walking, or calls a nap play.

For that reason we treat alignment not as a preprocessing convenience but as a property of the sample itself, recorded and inspectable, rather than assumed.

Three clocks that do not agree

Every source keeps its own notion of time. A wearable reports how long it has been awake rather than what hour it is. A stamp can be missing entirely. Encoded video adds its own constraint: a stream cannot be cut at an arbitrary instant, because playback has to begin from a picture the codec already stored, which on our ring segments lands roughly a second apart.

A tempting shortcut is to pad the window—take a few seconds either side of the wearable clip and trust that the interesting thing is somewhere inside. In practice padding imports a second room, a second motion and a second sound into the same labelled example, and it moves the visible start away from the recorded start. We removed padding and kept the window equal to the recording. The only slack left is the short head needed to begin on a stored keyframe, and that slack is written down rather than absorbed.

On the development hub a five-second collar recording aligned this way produced a preview whose video ran 5.558 s and whose audio ran 5.553 s, with 0.553 s of leading silence that matched the offset recorded alongside the sample to the millisecond. The clip ended exactly where the recording window ended. That is the property we care about: the number stored in the sample and the number a person would measure in the file are the same number.

The media a reviewer sees is the media that is learned from

Labelling pipelines often drift because a reviewer watches a convenience rendering while the training set is cut separately from raw sources. Two cuts mean two alignments, and only one of them was ever inspected by a human.

We therefore promote the reviewed artefact instead of re-cutting it. The aligned preview a person watched becomes the durable sample, byte for byte, and the alignment metadata travels with it. Reusing the existing camera ring rather than opening a second capture path is part of the same discipline: fewer independent renderings, fewer opportunities for two versions of one afternoon.

The change was introduced without invalidating earlier work. Samples labelled under the previous padded convention remain readable and playable as they were, and the several hundred already-labelled packages on the board were left in place rather than silently recut under new rules.

Absence is a measurement, not an empty field

Homes are not capture studios. The cat leaves frame. A segment of video ages out of the ring before anyone asks for it. A recording arrives without enough evidence to be placed on a wall clock. Motion data exists but is shorter than the model’s input window. A recording window straddles two video segments, so any single-segment cut would quietly truncate.

These are distinct facts about the world, and collapsing them into one blank makes “nothing” look like a single coherent class. The review surface keeps them separate—already saved, available now, still pending, rotated off the ring, missing a usable clock, or in error with the reason attached—so that a later analysis can ask why a sample is absent instead of only noticing that it is.

The same principle applies to human judgement. A reviewer is allowed to abstain, and abstention is stored as such. An uncertain sample is more useful than a confident wrong one, and a human mark sits beside what the system decided in production rather than overwriting it, so that disagreement between the two remains available for error analysis.

What this settles and what it does not

What the work establishes is narrow and load-bearing: within a sample, time, absence and provenance are explicit, and the artefact a human judged is the artefact a model consumes. That is a precondition for multimodal learning, not a result about behaviour recognition.

It says nothing yet about how much accuracy fusion buys over any single sensor, and it is not a public benchmark or an independently adjudicated clinical reference. Those claims require separate evidence. What alignment does buy is the right to interpret a later disagreement between camera and collar as a real disagreement about the cat, rather than as a bookkeeping artefact.