Physical Derivatives: Computing policy gradients by physical forward-propagation

(Supplementary material)