Network Resource Management

Objectives

This course provides modeling and operational tools to set up problems of resource management and optimal control of random processes operating on networked systems, illustrating several algorithmic solutions based on both stochastic optimization methods and reinforcement learning. Specific attention will be devoted to problem modeling, optimization techniques, performance evaluation, and computer simulations of the devloped algorithms.

Prerequisite: basic knowledge of calculus, linear algebra, and probability theory.

Final Exam: Oral exam (typically two open questions), plus a computer project carried out over one of the topics of the course.

Classroom code (2024-2025):   blj6woh  

Lessons: 

• Monday  11.00 - 14.00,  Room 25, S. Pietro in Vincoli. 

• Wednesday  18.00 - 20.00, Room 15, S. Pietro in Vincoli.   

Contents 

Part 1 - Service systems, scheduling, and dispatching  (Prof. Andrea Baiocchi)

• • Application context and performance indicators: Networked service system modelling. Main general performance indicators. Fundamental trade-offs among utilization efficiency, response time (or, age of information), energy consumption, and accuracy. Examples drawn from telecommunication networks, cloud computing, transportation systems, industrial processes.

  • Resource sharing: motivations and approaches. Scheduling algorithms and priority handling. Examples of strategic queueing. Scheduling optimization. Congestion and fairness. 

  • Network utility maximization: optimization problem statement, distributed solution, game-theoretic perspective.

References:  [1], [2], [3], [5], [6], [7], [8] 

Part 2 - Stochastic optimization and reinforcement learning  (Prof. Paolo Di Lorenzo)

• • Stochastic optimization of networked service systems: Randomized scheduling, Lyapunov optimization approach, min-drift plus penalty, virtual queues, power-stability trade-off.

  • Multi-armed bandits: action-value methods, incremental implementation, gradient bandit algorithms.

  • Reinforcement Learning: Markov decision processes, dynamic programming, policy evaluation and improvement. Temporal difference learning: Exploitation vs Exploration, Sarsa, Q-learning.

  • Examples of dynamic optimization applied to smart industry, 6G networks, edge computing, stock market trading. 

References:  [1], [3], [4]

Textbooks and resources:

[1]  Slides, notes, and codes

[2] Baiocchi, Andrea: Network Traffic Engineering - Stochastic models and applications. Wiley, 2020.

[3] Neely, Michael J. Stochastic network optimization with application to communication and queueing systems. Synthesis Lectures on Communication Networks 3.1 (2010): 1-211.

[4] Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. MIT press, 2018.

[5] Srikant, Rene, and Lei, Ying. Communication networks – an optimization, control and stochastic networks perspective. Cambridge University Press (2014): Ch. 1,2.

[6] Powell, W.B.: Reinforcement Learning and Stochastic Optimization: A Unified Framework for Sequential Decisions, Princeton Kelly, F. and Yuodvina, E.: Stochastic Networks. Cambridge University Press, 2014.

[7] Harchol-Balter, M.: Performance modelling and design of computer systems. Cambridge University Press, 2013.

[8] Srikant, R.: The mathematics of Internet congestion control, Birkhauser, 2003.

Last update: 23/02/2025