Resources

Özgür Şimşek

What is a useful action?

An agent's action space shouldn't stay fixed. It should get richer as the agent gains experience. Özgür gave three answers to what makes an action worth having. First, actions that move you between "modules" of the interaction graph: represent the agent–environment interaction as a graph, partition it with the Louvain algorithm to maximise modularity, and you get skills at several time scales, where the longer ones are built out of the shorter ones. Second, actions that carry you efficiently into unfamiliar territory: eigenvectors of a graph Laplacian reweighted by how often states have been visited produce a gradient that pushes you toward novel ones. Third, action sequences that recur in past successful behaviour, a "fracture" is a start state plus a short run of actions, and fractures from trained agents cluster tightly while random ones spread out almost uniformly. Cluster them, score them by how often and how widely they've been useful, and you have a skill hierarchy. Multi-Rooms, Taxi and Towers of Hanoi were some of the environments used as testbed.

Serena Booth

RLHF is survey design

Serena's argument is that RLHF is a survey, and the field runs it without the methodology that survey design has spent a century building. Reading through OpenAI's alignment publications from 2017 onward, she and her co-authors find RLHF doing double duty as survey and as content moderation, with normative choices baked in too deep to inspect from the outside. One example: labellers are screened for agreement with the researchers before their answers are ever collected. That's a sensible-sounding quality filter, but in survey terms it hand-picks the population, so the "human preferences" you end up measuring are the preferences of people selected for already agreeing with you. The reward model matters just as much as the sample. Preferences modelled as partial return don't pin down a unique reward function, while modelling them as regret does. Better still if the model you compute with is the one your raters are actually using, and it turns out you can nudge them toward it through transparency, training, or how the question is laid out on screen. Her broader finding is: people, experts included, are bad at designing reward functions. The trajectory alignment coefficient measures how bad, by correlating a stakeholder's ranking of trajectories with the one a candidate reward induces, no ground-truth reward required. The topic she left open: psychometrically validated surveys as a route to personalized reward functions.

Sara Aronowitz

Sequences of frames: the analogy with memory retrieval

Sara started from an observation: when we deliberate, we don't open with the higher-order question of what to think about. We interleave it with first-order ones. Her analogy is memory retrieval, which works the same way,  broad at first, then iterative, each recalled detail sharpening both confidence and the next question. Cooking is the everyday version: "What should I cook? -> What do I have already? -> Should I make something with lentils? -> Do I have the ingredients for dosas?" Why serialize at all? The easy answer is convenience, it's cheaper than solving the whole thing at once. But there are better ones. Retrieved details are used to retrieve other memories, and they re-orient you toward goals that may connect to, or entirely supersede, the one you started with. Going beyond convenience means asking whether the sequence is optimal for the underlying problem, and whether it's optimal for the agent's own interests in context. You want to reach the important things first, but importance isn't knowable before you've explored. Serialization is what lets the better question surface.

Joel Lehman

Does RL have a blind spot for Knightian uncertainty?

Frank Knight's distinction between measurable risk and genuine unknown unknowns is, Joel argued, exactly where RL's generalization story runs thin. That kind of uncertainty sits outside the field's core formalisms. Anticipate-and-train, where you collect diverse scenarios up front, only ever covers what you thought to imagine; zero-shot transferring a self-driving policy from the US to the UK is the kind of thing that still looks out of reach. So he looked to search processes that already cope with the unimaginable: human societies, and biological evolution, which produces agents that thrive in an open world without any explicit model of what might go wrong. The features worth borrowing are an open-ended search space, diversification paired with filtering, and an environment that keeps throwing up novel situations to filter against. The question he left open: what would an RL algorithm look like that handles unknown unknowns gracefully? He framed it as an opportunity rather than a complaint, RL is a good place to make conceptual progress on how any agent should wrangle Knightian uncertainty.