Partially-Observable Stochastic Hybrid Systems
Problem Formulation
This paper discusses the state estimation and optimal control problem of a class of partially-observable stochastic hybrid systems (POSHS).
The POSHS has interacting continuous and discrete dynamics with uncertainties.
The continuous dynamics are given by a Markov-jump linear system and
the discrete dynamics are defined by a Markov chain whose transition probabilities are dependent on the continuous state via guard conditions.
The only information available to the controller are noisy measurements of the continuous state.
Solution Approach
To solve the optimal control problem, a separable control scheme is applied: the controller estimates the continuous and discrete states of the POSHS using noisy measurements and computes the optimal control input from the state estimates. Since computing both optimal state estimates and optimal control inputs are intractable, this paper proposes computationally efficient algorithms to solve this problem numerically.
The proposed hybrid estimation algorithm is able to handle state-dependent Markov transitions and compute Gaussian- mixture distributions as the state estimates.
With the computed state estimates, a reinforcement learning algorithm defined on a function space is proposed. This approach is based on Monte Carlo sampling and integration on a function space containing all the probability distributions of the hybrid state estimates.
Scheme of the separable controller for the POSHS
Simulations
Finally, the proposed algorithm is tested via numerical simulations. The two spacecraft A and B are getting close to each other with horizontal velocity V_A and V_B parallel to each other. Both Spacecraft A and B are equipped with thrust nozzles.
The goal of optimal control is to design a control law for Spacecraft A so that the distance x between the horizontal velocities V_A and V_B is minimized. While the controller for Spacecraft A’s thrust nozzle does not have accurate information about Spacecraft B’s action (whether Spacecraft B is using thrust), the controller has to estimate Spacecraft B’s action by using a noisy measurement of x, then decide Spacecraft A’s action accordingly.
Evolution of the POSHS under closed-loop control (the left figure)
the POSHS's state trajectory, noisy observations, and hybrid estimation results are shown.
The simulation results shown in the left figure demonstrate that:
the closed-loop system's stability is achieved and
the distance between the two spacecraft converges to a stable distribution around zero.
The estimated discrete state probability and the control inputs to the system (the right figure)
In most of the time, the control inputs tend to compensate for the current estimated motion of Spacecraft B so that the distance x between Spacecraft A and B is minimized.
Evolution of the POSHS under closed-loop control
Estimated discrete state probabilities and control input to Spacecraft A
Related Publication
W. Liu and I. Hwang, “Partially-Observable Stochastic Hybrid Systems (POSHSs) State Estimation and Optimal Control,” Asian Journal of Control, March 20, 2015, DOI: 10.1002/asjc.1121