The Data

Dataset overview

This challenge is built on the Berkeley Open Extended Reality Recording Dataset 2023 (BOXRR-23), created by researchers at UC Berkeley.

BOXRR-23 contains 4,717,215 motion-capture recordings generated by 105,852 real users of extended reality devices. The Berkeley team sourced these recordings from broadly public sources, converted them into a single format, combined them with metadata from open-access APIs, removed identifiable user data, and packaged the result for the research community.

The dataset totals 4.7 TB compressed, expanding to over 8.0 TB raw, and is distributed in 106 chunks of up to 1,000 users each, averaging 45 GB per chunk.

→ Dataset home: Berkeley Open Extended Reality Recording Dataset 2023 (BOXRR-23) | Berkeley RDI

You do not need all of it. You need the right slice of it, and the pipeline that produces that slice is part of what you are being asked to build. The dataset is hosted on Huggingface, and you will be able to download the data by sessions and keep only what is necessary.

Data filtering scripts

Constructing the training corpus is part of the challenge. Participants are expected to:

1. Develop download scripts that retrieve only the sessions meeting the sweep and velocity criteria below.

2. Filter by sampling rate — keep only data within 85–110 Hz, and tag it as 100 Hz data.

3. Preprocess — quaternion normalisation, skipping sessions with too few samples, interpolating missing data within a session.

4. Produce an HDF5 file of the cleaned dataset.

The original XROR recordings are sanitised down to what this task actually requires: normalised head-motion quaternions and their timestamps. Everything else is discarded.

For reference, STMicroelectronics' internal pipeline applied to these criteria yields a single HDF5 file of approximately 40 GB. The key is to keep a session only if it satisfies all three of the following.

1. Sampling rate: 85–110 Hz: Estimate the rate from the median inter-sample interval across the session. Sessions inside this band are retained and tagged as 100 Hz data. Sessions outside it are discarded — not resampled from a distant rate.

2. Average sustained median sweep ≥ 17°: Sweep captures how much ground the head actually covers.

3. Average sustained median velocity ≥ 15°/s: Velocity captures whether the head is genuinely in motion.

Together these two thresholds exclude the large fraction of recordings in which a user is essentially stationary. A model trained on still heads learns to predict stillness, which is both easy and useless.

Data filtering tutorial during downloading:

For each candidate session downloaded from huggingface

Step 1 — Trim. Discard the first 5 seconds of each session. Headset tracking is unreliable at startup, and the opening seconds routinely contain jitter that would inflate both measures.

Step 2 — Window. Divide the remainder into non-overlapping 5-second windows. Skip any window containing fewer than 10 valid samples. A session with fewer than 20 samples after trimming is not assessed at all.

Step 3 — Convert to Euler angles. Convert the normalised quaternions to xyz Euler angles, in degrees. Both measures are computed independently on each of the three axes, so that a session with vigorous yaw but little pitch still qualifies on the strength of its yaw.

Step 4 — Sweep, within one window, on one axis. The angular extent covered, taken as the 10th-to-90th percentile range:

sweep = percentile(angle, 90) - percentile(angle, 10)

The percentile range is used rather than the full minimum-to-maximum range so that a single spurious sample cannot inflate a window.

Step 5 — Velocity, within one window, on one axis. The total angular path travelled per second:

diffs    = absolute differences between consecutive angle samples

wrapped  = 360 - diff   where diff > 180

         = diff         otherwise

velocity = sum(wrapped) / window duration in seconds

The wrap is essential. Without it, every crossing of the plus-or-minus 180 degree boundary registers as a spurious jump of more than 300 degrees, and near-motionless sessions pass the filter.

Step 6 — Aggregate to the session. For each axis, take the median across all windows:

S[axis] = median over windows of sweep

V[axis] = median over windows of velocity

This median is what makes the measures sustained. A session qualifies because the head moves persistently, not because of one dramatic burst in an otherwise still recording. A global min-to-max sweep, or a session-wide mean velocity, would admit precisely the recordings these thresholds exist to exclude.

Then take the maximum across the three axes:

S = max(S[x], S[y], S[z])

V = max(V[x], V[y], V[z])

Step 7 — Threshold. Retain the session if both conditions hold:

S > 17     (degrees)

V > 15     (degrees per second)

Step 8 — Keep only data within 85 to 110 Hz, and tag it as 100 Hz data. Estimate each session's rate from its inter-sample intervals — the median interval is a robust choice. Sessions falling outside the band are discarded, not resampled from a distant rate.

Repeat the steps for all sessions being downloaded from Huggingface.

Other required preprocessing

• Quaternion normalisation. Scale every quaternion to unit norm. Reject samples that are non-finite, or whose norm is too close to zero to normalise stably.

• Skip sessions with too few samples. A single training example consumes 400 context samples plus 20 targets, so 420 is the absolute floor. Set your threshold well above it, so that each retained session yields a useful number of distinct windows.

• Interpolate missing data within a session. Use spherical linear interpolation (SLERP). Choose a maximum gap you are prepared to fill, and split the session rather than interpolating across longer dropouts; half a second of invented motion is worse than a shorter session.

• Produce an HDF5 file of the cleaned dataset. Structure it however suits your loader.

Recommended preprocessing include:

• Put the data on a uniform 100 Hz grid. The task is specified in samples — 400 of context, 20 of prediction — and that corresponds to 4 seconds and 100 milliseconds only if the samples are evenly spaced. The evaluation stream is uniform. Training on irregular timestamps means training under a different regime from the one you are scored in.

• Align quaternion hemispheres. Because q and -q encode the same rotation, raw recordings frequently flip sign between consecutive samples. This produces a large discontinuity in the 4-D signal at a point where the underlying rotation is perfectly smooth. Walk each session in order and negate any sample whose dot product with its predecessor is negative. Without this step, a model regressing on raw quaternion components spends capacity fitting artefacts that do not exist in the motion.

Building validation splits

The scored test set comes from people and hardware absent from BOXRR-23. Your local validation should imitate that, or it will mislead you. 

• Split by user, never by window. Windows drawn from the same session overlap almost entirely and are near-duplicates of one another. A random window split leaks the validation set into training and produces numbers far better than your eventual score.

• Fix your averaging convention. Sampling a handful of windows per session gives a session-weighted figure with high variance; averaging across all windows gives a window-weighted figure. Both are defensible, and they are not the same number. State which you use and stay with it, or you will chase regressions that were never there.

• Report a constant-velocity baseline. Extrapolating the current angular velocity forward 20 samples is a strong, parameter-free predictor at a 100 millisecond horizon. If your model does not clearly beat it, its parameters are not earning their place — and finding that out early is worth a great deal.

• Do not validate on the selection measures. Sweep and velocity decide which sessions enter your corpus. Models are scored on quaternion geodesic error. They are different quantities and should not be confused.

Practical tips: feeding a 40 GB corpus to a GPU

A cleaned corpus of this size will not fit in GPU memory, and naive random access to an HDF5 file will leave your GPU idle most of the time. The following patterns are what make training throughput acceptable.

The core problem. Your dataset is millions of overlapping windows spread across tens of thousands of variable-length sessions. Randomly sampling one window at a time means one HDF5 read per sample, each decompressing a chunk to extract a few hundred rows. That is catastrophically slow, and it is the single most common reason a training run reports low GPU utilisation.

Solutions:

• Chunk at the session level, not the window level. Group sessions into memory chunks of a few gigabytes — 3 GB is a reasonable starting point. Load one chunk into host RAM, generate every window from it, then move to the next chunk. A session of N rows with four float32 quaternion components costs 16N bytes, so you can size chunks by simple arithmetic before reading anything.

• Shuffle at three levels, not one. Shuffle the session order once at startup, shuffle the order in which chunks are consumed each epoch, and shuffle the window start indices within each session. This approximates global shuffling closely enough for training while preserving sequential reads. Shuffling only one of the three leaves strong temporal correlation inside each batch.

• Read each session contiguously, once. Slice the whole session out of the HDF5 in a single read, then window it in memory. If you compressed the file, this matters even more: gzip chunks are decompressed on every touch, so touching a session once and reusing it thousands of times amortises the cost to nothing.

• Generate windows as strided views, not copies. Once a session is a contiguous tensor, tensor.unfold(0, window_length, 1) produces every sliding window as a zero-copy view. Materialising them as separate arrays multiplies your memory footprint by the window length for no benefit.

• Do the windowing on the GPU. Move an entire session — or an entire chunk, if it fits — to the GPU as one transfer, then index batches on-device. One large host-to-device copy is far cheaper than thousands of small per-sample copies, and the slicing itself becomes free. This is usually worth more than any other single optimisation.

• Shard correctly across dataloader workers. If you use a streaming or iterable-style dataset, each worker receives a full copy of the iterator and will otherwise emit the same data. Partition chunks by worker index — for example, assign chunk i to worker i % num_workers — or you will silently train on duplicated batches. Enable pinned memory, keep workers persistent between epochs, and use non-blocking transfers so the next chunk uploads while the current batch trains.

Data use and citation

BOXRR-23 is distributed by UC Berkeley under its own terms. Participants are responsible for reading and complying with the licence and access conditions published on the Berkeley RDI dataset page.

Papers submitted to this challenge must cite BOXRR-23, using the citation given on that page. Recordings are de-identified: do not attempt to re-identify users, and do not redistribute raw or derived data outside your team.

Related publications using this dataset are listed on the Berkeley RDI dataset page and challenge home page.