Introduction to discrete-time reinforcement learning control in Complex Engineering Systems

Within the context of the Complex Engineering Systems (CES), this editorial aims to describe the recent progress of discrete-time reinforcement learning (RL) control.

Considering the widespread application of the digital computer in the CES, which process the data in discrete-time form, and the nonlinearity inherent in the engineering, the nonlinear discrete-time control design has garnered growing attention in the modern control engineering. For instance, when the backstepping method is employed to design the controller for a nonlinear discrete-time system, it may suffer from a noncausal problem. As the authors claimed, the causality contradiction problem may arise^[1], where the future signal is embedded in the current control signal, leading to controller design failure. To solve this problem, various system transformations were developed, making them one of the perennial topics for nonlinear discrete-time systems.

The control design is one of the most important topics of CES; as an optimal control strategy, RL has received increasing attention. It can not only make a compromise between the control cost and performance but also decrease the impact of external environment through continuous exploration^[2]. Many RL techniques exist, such as Q-learning, adaptive dynamic programming (ADP), policy iteration, etc. Among them, actor-critic is one of the classical RL techniques, which is simple and easy to apply. However, it should be noted that the gradient descent method is employed to learn the weight vector that searches for the optimal solution from a single point, and it may easily fall into the local optimal. Therefore, solving the local optimal problem is one of the hot topics in the RL control design. In the following section, the above control topics of research progress will be introduced.