Detailed Tutorial Outline

The tutorial begins with an introduction to the foundational concepts required for NeSyRL. For Reinforcement Learning (RL), this includes policies, value functions, and key deep RLarchitectures (e.g., 𝜀-greedy, actor-critic). For Knowledge Representation (KR), we will formalize symbolic representations of states, actions, policies, preferences, and constraints, and introduce the basics of logic programming for planning. We will conclude this module with a critical overview of the advantages and limitations of neural vs symbolic AI, motivating the need for their integration. 

The core part of the tutorial will analyze three main strategies for NeSyRL, introducing both the theory and selected example applications from the fields of single and multi-agent planning and robotics. 

We will first introduce reward machines as one of the most established approaches for incorporating high-level task structure into RL. This part will explain how reward machines use automata-based symbolic representations to encode non-Markovian reward functions, how they are composed with the environment to guide policy learning, and why they are effective for sequential tasks and sparse-reward settings. 

We will also introduce the similar framework of restraining bolts, as a modular paradigm for controlling agent behavior through external supervisory mechanisms, useful for runtime policy monitoring and guidance. 

The tutorial will then move to NeSyRL integration for safety, a crucial requirement for single and multi-agent systems, especially robots. Specifically, we will present probabilistic logic shields, which extend earlier shielding concepts by accounting for uncertainty in both the environment and the agent’s beliefs. This module will describe how safety and correctness requirements can be expressed in highly expressive logical formalisms and enforced at runtime through probabilistic reasoning, allowing learning agents to operate safely while still exploring and optimizing performance at the neural level. 

Finally, the tutorial will present more advanced NeSyRL integrations, acting directly at the algorithmic level of RL. Specifically, we will present NeSyRL frameworks designed for model-based RL and 𝜀-greedy RL. These algorithms exploit partial (potentially imperfect) symbolic policies, defined for small domains and encoded as symbolic programs, to perform automated reasoning and compute promising (or inconvenient) actions. These suggestions can then be generalized to more complex domains, serving as probabilistic exploration bias or exploitation priors. These approaches enable transparent reasoning, improved sample efficiency, and generalization to more complex domains. 

The tutorial will conclude by summarizing key takeaways and highlighting open research challenges in NeSyRL, including algorithmic integration with more complex RL architectures and symbol learning and grounding. Participants will be encouraged to discuss potential real-world applications, drawing on their expertise, to stimulate community growth and define pathways for advancing neurosymbolic decision making.