A Reinforcement Learning Method for Continuous Domains Using Artificial Hydrocarbon Networks

Hiram Ponce, Guillermo González-Mora, Lourdes Martínez-Villaseñor

Abstract - Reinforcement learning in continuous states and actions has been limitedly studied in occasions given difficulties in the determination of the transition function, lack of performance in continuous-to-discrete relaxation problems, among others. For instance, real-world problems, e.g. robotics, require these methods for learning complex tasks. Thus, in this paper, we propose a method for reinforcement learning with continuous states and actions using a model-based approach learned with artificial hydrocarbon networks (AHN). The proposed method considers modeling the dynamics of the continuous task with the supervised AHN method. Initial random rollouts and posterior data collection from policy evaluation improve the training of the AHN-based dynamics model. Preliminary results over the well known mountain car task showed that artificial hydrocarbon net-works can contribute to model-based approaches in continuous RL problems in both estimation efficiency (0.0012 in root mean squared-error) and sub-optimal policy convergence (reached in 357 steps), in just 5 trials over a parameter space θ ∈ R86.

Data from experimental results are available at: http://sites.google.com/up.edu.mx/reinforcement-learning/.

Accepted on World Congress on Computational Intelligence - International Joint Conference on Neural Networks (WCCI - IJCNN 2018)

Design and Equilibrium Control of a Force-Balanced One-Leg Mechanism

Hiram Ponce, Mario Acevedo

Abstract - The problem of equilibrium is critical for planning, control, and analysis of legged robot. Control algorithms for legged robots use the equilibrium criteria to avoid falls. The computational efficiency of the equilibrium tests is critical. To comply with this it is necessary to calculate the horizontal momentum rotation for every moment. For arbitrary contact geometries, more complex and computationally-expensive techniques are required. On the other hand designing equilibrium controllers for legged robots is a challenging problem. Nonlinear or more complex control systems have to be designed, complicating the computational cost and demanding robust actuators. In this paper, we propose a force-balanced mechanism as a building element for the synthesis of legged robots that can be easily balance controlled. The mechanism has two degrees of freedom, in opposition to the more traditional one degree of freedom linkages generally used as legs in robotics. This facilitates the efficient use of the projection of the center of mass criterion with the aid of a counter rotating inertia, reducing the number of calculations required by the control algorithm. Different experiments to balance the mechanism and to track unstable set-point positions have been done. Proportional error controllers with different strategies as well as learning approaches, based on an artificial intelligence method namely artificial hydrocarbon networks, have been used. Dynamic simulations results are reported.

Videos of experiments will be available at: http://sites.google.com/up.edu.mx/smart-robotic-legs/ .

In review on International Conference on Intelligent Robots and Systems (IROS 2018)

Dynamic Modelling and Balancing

Mario Acevedo, Hiram Ponce

Abstract - Multi-body dynamics has been a fundamental tool for modeling, simulation and design of human-like locomotion systems. Either in the prosthetic and orthotics sector to develop devices for improvement or restoration of mobility, or as in the simulation, design, and optimization of humanoid robots. A lot of research and development has been done in these challenging areas where new mechanisms and improvements in dynamics are always welcome. The dynamic balancing of mechanisms (force and moment balancing at the fixed base) is an area that, along with multi-body dynamics, can help to improve the design of human-like locomotion systems. In this chapter the application of a force balanced mechanism is proposed as a leg to be part of a biped robot. Stability is analyzed through the application of learning approaches based on an artificial intelligence, namely artificial hydrocarbon networks. Modeling and results from multi-body dynamics simulation are presented.

To be published on Design and Operation of Human Locomotion Systems (Marco Ceccarelli, Guiseppe Carbone, Eds.), Elsevier, 2018.