My work on average-reward RL targets stable critic learning for continuing decision problems. I combine constrained optimization with a neural correction term and integrate the update into PPO as Single-Chain PPO (SCPPO).
The critic converged faster and more stably in 14 out of 15 OpenAI Gym environments, In MuJoCo, SCPPO improved HalfCheetah return by 13.5% over PPO with FP32 critics; on continuing Hopper, the FP16 performance drop fell from 41.4% to 2.6%.
I built a continuing Franka manipulation benchmark in Isaac Lab and modified PPO's advantage estimation using imagined post-delivery holding continuations. In a single-seed, state-based evaluation across 128 simulated arms, the policy achieved 3.53 times the throughput of ordinary GAE.
I then integrate stereo CNN localization, calibrate triangulation, and temporal FIFO tracking with the frozen PPO policy, using robot state and delivery everts. In one 60-second visual-control simulation, the arm completed 33 cube deliveries without resets, with 8.70 mm mean 3D target-position error.