World Models from Scratch Series

Action Items

• [ ] Nahid to publish GitBook and share GitHub + GitBook links in the Cohere channel after the session

• [ ] Nahid to post session links on Twitter and LinkedIn

• [ ] Mayank Bhaskar to share the Cohere Labs community join link in the chat

• [ ] Mayank Bhaskar to publish a written summary of this session (volunteered, as the session was not recorded)

• [ ] Attendees to secure GPU access (Modal $30 starter credits, Lightning, or Google Colab) before the hands-on session in ~4 weeks

Session Overview & Logistics

• Hosted by Nahid, Nahid, an AI Researcher (https://x.com/nahidalam)

• This is a bi-weekly series of ~10 chapters covering world models from theory to hands-on implementation

• Attendees who complete ≥75% of sessions will receive a certificate from Cohale Labs

• The session was not recorded; Mayank Bhaskar volunteered to provide a written summary

• GitBook (open, interactive) will be published publicly after this session ends; GitHub repo is already public

• Chapters 4 and 5 are expected to generate the most interest; earlier chapters build foundational concepts

Chapter 1: Core Concepts

• What is a "world"? A bounded environment — not necessarily visual or video-based

  • Examples: self-driving cars, drones, computer-use agents (clicking through Slack), Excel-operating agents in banking

• What is a world model? A learned system that models how an environment changes — predicting possible futures given current state and actions

  • Key distinction: world model ≠ video model; video is just one possible world type

• Key vocabulary defined:

  • Observation — what the world looks like now (e.g., camera frames, sensor data)

  • State — encoded representation of the current situation

  • Action — input that drives the agent (e.g., braking, speed change)

  • Transition — one predicted step from current state to next

  • Rollout — a full sequence of transitions representing one possible path to a goal

• Multiple futures: A model naturally produces multiple probable next states; the highest-probability outcome is typically selected, but all possible futures can be explored

• Dynamics are environment-specific (e.g., drone aerodynamics, car vehicle physics, biological processes)

• Why world models? Core value is reasoning over possible futures — especially useful for rare/edge-case scenarios (e.g., a child unexpectedly crossing the road, or simulating impact of a new medicine on a geography)

• Simulators: Historically rule-based; increasingly integrated with learned world models and RL. Seen as a space ripe for disruption, especially for startups

Nahid Alam (@nahidalam) on X

Building | Research Resident @PrimeIntellect | Maya (https://t.co/ebaV9lpXU6) | @litehouseio | Alum @backstage_cap, @meraki, @intel, #TEDx @TBSNetwork

X

Chapter 2: Hands-On Demo (NVIDIA Cosmos Model)

• The series will use the NVIDIA Cosmos ~2.5B parameter model for inference; a similar model will be built in Chapter 4 (capped at ~3B parameters)

• Demo showed a rollout explorer using a sand-mining input video

  • Steps: load video → encode frames → load Cosmos model → generate one or multiple rollouts using different seeds

    • A disagreement map can highlight where two rollouts diverge

    • Key parameter: guidance scale — controls how strongly the prompt influences output (lower = less controlled)

• Pre-generated outputs are committed to the GitHub repo so attendees without GPUs can follow along

• Tested on NVIDIA A40 (48 GB VRAM); inference may be possible on smaller GPUs; Mac GPU not tested and likely challenging

Q&A Highlights

• Is the world model generative? Yes — the series uses generative models similar to transformers

• Can a rollout be categorized as continual learning? Unanswered — flagged as a good open question

• Is it related to graphs? Potentially — the tree-like rollout structure is graph-like in CS terms, but no direct graph-world-model work was cited

• World model vs. tabular RL? World models handle complex, high-variation environments; tabular/rule-based methods suffice for deterministic, low-variation scenarios

• How is uncertainty represented? Through the probabilistic nature of the learned system (e.g., Markov models, LLM-based approaches); probability is the representation of uncertainty

• Where is causality embedded? Likely in actions (direct events) and in environment-specific dynamics, possibly also trickling through the loss function — noted as an interpretability question

• World model vs. RL? RL is a type of learning; world model is a type of model — RL can be used to train a world model but is not required

• Can inference run on Google Colab? Likely yes for inference (Chapters 2–3); training may be limited depending on Colab tier

GPU Resources for Upcoming Hands-On Sessions

• Modal — $30 free compute credits on the starter plan

• Lightning AI — previously offered free credits; current status unclear

• Google Colab — works but may require workarounds for memory constraints

• Hands-on session (Chapter 4 build) is approximately one month away; attendees encouraged to prepare GPU access in advance