Developing Foundation Models for Head Pose Estimation 

Challenge Goal

The goal of the challenge is to develop foundation models that can: 

• Accurately predict n samples of future probable head pose given a window x of context quaternion samples. 

• Generalize to any user zero-shot without fine-tuning. 

• Deployable on commercial XR edgeAI platforms. 

More specifically, the model must be as follows:

• Input: Up to 400 head-motion quaternion samples (4 seconds of context at 100 Hz)

• Output: The next 20 quaternions — 100 milliseconds of future head pose

• Mode: Streaming: predict 20, receive 1 new sample, slide the window by 1, predict again

• Requirement: Zero-shot generalisation to any user, with no fine-tuning

• Target: Deployable on commercial XR edge-AI platforms

Your model must be accurate, small, and fast — all three are scored.

The dataset

This challenge is built on BOXRR-23 (Berkeley Open Extended Reality Recording Dataset), the largest motion-capture dataset of its kind: 4,717,215 recordings from 105,852 real XR users, gathered by UC Berkeley from broadly public sources, de-identified, and unified into a single format.

Since the dataset is huge (4.71 TB in size) and not all sessions are useful, STMicroelectronics expects participants to develop scripts that download only the sessions worth learning from — those with an average sustained median sweep of 17° and an average sustained median velocity of 15°/s — filtered to a 85–110 Hz sampling range and tagged as 100 Hz. The sanitised result is a single HDF5 file of normalised head-motion quaternions and timestamps (hint: the final HDF5 file is only a few GB in size)

Reference Papers on the dataset should be cited. 

• Nair, Vivek, et al. "Berkeley open extended reality recordings 2023 (BOXRR-23): 4.7 million motion capture recordings from 105,000 XR users." IEEE Transactions on Visualization and Computer Graphics 30.5 (2024): 2239-2246. 

Evaluation — fair by design

STMicroelectronics will evaluate every submission on an unreleased test set that is not part of BOXRR-23. The test set is never distributed to participants.

This is a deliberate departure from challenges that hand out the test data, where fitting to the test set is always a temptation. Here, the only way to score well is to genuinely generalise.

Submissions are ranked on four criteria — prediction accuracy (40%), parameter count (10%), latency on an XR platform (30%), and innovation in architecture, compression, or training technique (20%).

Important Dates and Milestones

• Challenge opens: Sep 15, 2026

• Tutorial  1: Oct 4, 2026

• Tutorial 2: Oct 12, 2026

• Challenge registration closes: Oct 27, 2026

• Submission of model and paper: Nov 15th, 2026

• Reviews sent to participants: Nov 20, 2026

• Camera-ready papers: Dec 8, 2026

• Conference: Jan 19, 2027

What you deliver

• A trained model — ONNX, PyTorch, or TensorFlow — with documentation on how to load it and run inference on a standard Linux machine with CUDA support.

• Either 4-6 pages in IEEE format or 8-10 pages IJABC format scientific paper describing your work.

• At least one team member presents at the conference.

Contact

Swapnil Sayan Saha, Software Design and Development Staff Engineer

STMicroelectronics Inc.,

swapnilsayan.saha@st.com AND swapnilsayansaha@ieee.org (make sure your query goes to both emails)