RSPSO Algorithm for Finding the Best Point for Robot to Score a Goal


  • Asseel Jabbar Almahdi University of Thi-Qar, Computer Science and Mathematics Collage, Computer Science Department


Reinforcement learning, Particle swarm optimization (PSO), robot soccer, Q-learning


The issue of robotic football game is one of the most complex multi-agent systems. In order to make the
game more enjoyable and exciting by scoring goals, therefore it's important for a robot football agent to
have a technique on how to score a goal. This paper proposes RSPSO algorithm, where the particle swarm
optimization algorithm (PSO) based on Q-learning is used to find the best point from the goal gate to
shoot the ball toward it. As we show in simulation, our method enables the robot to learn soccer skills to
score a goal.


Kitano, H., Asada, M., Kuniyoshi, Y., Noda, I., Osawa, E., Robocup, H.M.: "A challenge problem

for ai. AI mag.", 18(1), 73, 1997.

Visser U & Burkhard HD, "RoboCup: 10 years of achievements and future challenges", AI

Magazine, 28(2), p. 115, 2007.

A. Likas, "Multivalued Parallel Recombinative Reinforcement Learning: A Multivalued Genetic

Algorithm", In Proc, HERCMA Conference, 1998.

Watkins CJCH," Learning from Delayed Rewards", Doctoral Dissertation, University of Cambridge

(MA), 1989.

Budden DM," Applications of machine learning techniques to humanoid robot platforms", Doc-toral

Dissertation, University of Newcastle, Australia, 2012.

Ogino M, Katoh Y, Aono M, Asada M & Hosoda K, "Reinforcement learning of humanoid rhythmic

walking parameters based on visual information", Advanced Robotics, 18(7), pp. 677-697, 2004.

Kobayashi H, Osaki T, Williams E, Ishino A & Shinohara," A, RoboCup 2006: Robot Soccer World

Cup X", Springer, Berlin, Germany, Available from,

Riedmiller M & Gabel T, "On experiences in a complex and competitive gaming domain:

Reinforcement learning meets RoboCup", Proceedings of the Computational Intelligence and Games,

CIG 2007, IEEE Symposium, pp. 17-23, 2007.

Duan Y, Liu Q & Xu X, "Application of reinforcement learning in robot soccer", Engineering

Applications of Arti_cial Intelligence, 20(7), pp. 936-950, 2007.

Riedmiller M, Gabel T, Hafner R & Lange S," Reinforcement learning for robot soccer",

Autonomous Robots, 27(1), pp. 55-73, 2009.

Hester T, Quinlan M & Stone P, "Generalized model learning for reinforcement learning on a

humanoid robot", Proceedings of the IEEE International Conference on Robotics and Automation

(ICRA), pp. 2369-2374, 2010.

Stone P, Sutton RS & Kuhlmann G, "Reinforcement learning for RoboCup soccer keepaway,

Adaptive Behavior", 13(3), pp. 165-188, 2005.

Neri JRF, Zatelli MR, Farias dos Santos C & Fabro JA, "A proposal of Q-learning to control the

attack of a 2D robot soccer simulation team", Proceedings of the Robotics Symposium and Latin

American Robotics Symposium (SBR-LARS), pp. 174-178, 2012.

E. Uchibe. "Cooperative Behavior Acquisition by Learning and Evolution in a Multi-Agent

Environment for Mobile Robots". PhD thesis, Osaka University, January 1999.

P. Stone and M. Veloso. "Team-partitioned, opaque-transition reinforcement learning". In M.

Asada and H. Kitano, editors, RoboCup-98: Robot Soccer World Cup

II. Springer Verlag, Berlin, 1999. Also, in Proceedings of the Third International Conference on

Autonomous Agents,1999.

T. Andou. "Refinement of soccer agents' positions using reinforcement learning". In H. Kitano,

editor, RoboCup-97: Robot Soccer World Cup I, pages 373-388. Springer Verlag, Berlin, 1998.

M Yoon, J Bekker, S Kroon, "New reinforcement learning algorithm for robot soccer", ORiON

journals, Volume 33 (1), pp. 1-20, 2017.

Jo˜ao Cunha · Rui Serra · Nuno Lau., " Batch Reinforcement Learning for Robotic Soccer Using

the Q-Batch Update-Rule", Springer Science+Business Media Dordrecht, J Intell Robot Syst, DOI

1007/s10846-014-0171-1, 2015.

HAOBIN SHI, ZHIQIANG LIN, KAO-SHING HWANG," An Adaptive Strategy Selection

Method with Reinforcement Learning for Robotic Soccer Games", IEEEAccess, VOLUME 6, 2018.

Kennedy, J.; Eberhart, R. "Particle Swarm Optimization". Proceedings of IEEE International

Conference on Neural Networks IV. pp. 1942– 1948, (1995).