Reinforcement learning-based control for offshore crane load-landing operations

1. INTRODUCTION

Setting a heavy object down on the deck of a vessel is one of the most common marine operations. During the load-landing or -lifting process, any disturbance such as heave motion may lead to a significant impact on the load and equipment, which may cause fatal injury to the crew and permanent damage. Due to a variety of factors, such as ship motions, crane mechanics, and other factors, achieving a soft load landing with acceptable impact force and a small distance is challenging. Hence, a lot of efforts have been carried out to facilitate this operation; some introduced a variety of control algorithms to automate the process, and others intended to provide training methodology for the operators through simulators, virtual reality, and augmented reality. Several control techniques for marine operations have been proposed. A payload position control of offshore crane was developed by Park et al.^[1] using uniformly ultimately bounded (UUB) theory, integrated with the input–output linearization control technique (IOLC). A passive heave compensation system for the offshore landing process was studied by Huster et al.^[2] and Ni et al.^[3]. The passive shock isolator has some limitations due to the higher weights of loads and the hard working environment; hence, Zhu et al.^[4] presented a feedback control strategy where a shock isolator with an optimal controller is introduced to reduce and minimize the peak force transferred to the load. The commonality of the previous techniques is that they are model-based control algorithms. Hence, different dynamical models are presented in the literature. Mackojc et al.^[5] presented a six-degree of freedom (6DOF) mathematical model for the vessel and a 3DOF model for the lifting system, and they combined them to produce a payload–vessel system to facilitate the comprehensive investigation of mutual interactions. Idres et al.^[6] and Ellermann et al.^[7] developed a nonlinear coupled model for a crane and cargo based on the assumption that the cargo was a point mass. Cha et al.^[8] established the coupled model between the floating crane and the cargo considering the geometries of cargo. Spong's book^[9] and Williams et al.^[10] built a robot-like model for the knuckle crane using DH notation and Lagrange's approach. Another approach is using the reinforcement learning (RL) theory. RL is a learning method that gradually explores the optimal policy by interacting with the environment^[11]. Andersson et al.^[12] proposed actuator space control policies for a forestry crane manipulator to be energy efficient in log grasping by involving a simple curriculum in a deep reinforcement learning setup. Moreover, Gaudet et al.^[13] introduced a new integrated guidance and control method for a spaceship based on reinforcement learning concepts. They used the control algorithm to learn a policy that directly maps the lander's estimated state to a commanded thrust for each engine, resulting in precise and fuel-efficient trajectories. Sun and Xie^[14] introduced a backstepping control scheme based on reinforcement fuzzy Q-learning to control container cranes. Ding^[15] built different environments and used deep neural networks with RL algorithms to set down the load softly. Based on the aforementioned results, in this work, we focus on providing optimal control sequences for an offshore crane to get acceptable impact velocity while landing its load on a moving vessel using reinforcement learning algorithms, where a mathematical model of the offshore crane was established to be involved in the reinforcement learning environment, Q-learning algorithm was created using MATLAB platform to control the crane actuators, and the algorithm performance was measured against the variation of its hyperparameters. Moreover, the bias that usually occurs in Q-learning algorithms was tested using Double Q-learning algorithm. Double Q-learning is an off-policy value-based reinforcement learning algorithm with no positive bias in estimating the action values in stochastic environments.

The remainder of this paper is organized as follows. In Section 2, the problem statement with the operation system assumptions are described in detail, and the algorithms' structures are provided as well. In Section 3, a description of the RL algorithms that are used during this work is given. In Section 4, a simulation for the control sequences is demonstrated, and a comparison between the behavior of the environment under different assumptions is introduced. Section 5 is a discussion of the obtained results, and it stands on the strength of this work.

2. PROBLEM DESCRIPTION

2.1. Environment

In this work, two different environments are considered, namely initial environment and upgraded environment, as described in the following.

2.1.1. Initial environment

In this environment, the wave of every episode is randomly generated from the JONSWAP spectra created by the same sea state. Therefore, the wave elevation differs between episodes but not the statistical properties ^[16]. Here, the sea state is assumed to be:

$$ H_s = 1.5 m $$

$$ T_p = 30 s $$

where H_s is the wave height and T_p is the peak wave period. In a real application, the vertical motion of a vessel can be predicted approximately 4 s ahead of time with high accuracy^[17]. This gives us the reliability to plane a control sequence that can be visualized to the crane operator using technologies such as augmented reality. Moreover, the following assumptions are imposed on the load-landing operation problem:

Assumption 1

Ⅰ Neglect all the crane and load dynamics.

Ⅱ The mass and stiffness of the barge are neglected.

Ⅲ The barge has the same amplitude of the wave (barge dynamics is neglected and its response is 1 to all the wave frequency).

Ⅳ Neglect the wave effect on the crane.

Ⅴ Consider the hoist speed as the action to the control input directly.

State observation

In this environment, the agent–environment interaction is described through the following set of equations:

(1) $$ \begin{equation} P_h(i+1)=P_h(i)+a_t \cdot t_{step} \end{equation} $$

(2) $$ \begin{equation} D_r(i+1)=D_r(i)-\lvert P_w\rvert \end{equation} $$

where P_h is the hoist position; P_w is the wave height; D_r is the relative distance between P_h and P_w; a_t is the chosen action (hoist velocity); and t_step is the time step of discretization. The action space in this environment consists of 11 actions $$ [(-5 / 30):(5 / 30)] $$ with step (1 / 30) m/s, which are the hoist velocities^[15].

2.1.2. Upgraded environment

The environment is upgraded by inserting the forward kinematic model of a knuckle crane. Considering the crane as a three-revolute joints robotics arm, the input to the environment, in this case, is the actuators' angles.

State observation (forward kinematic)

The mathematical model for the knuckle crane was generated by simulating the crane as a robotic arm with three joins, as shown in Figure 1, with the following assumptions:

Figure 1. Crane kinematic structure.

Assumption 2

Ⅰ Neglect the dynamics of the load and wires.

Ⅱ The second and third joints are actuated through hydraulic cylinders, but the actuator models are not taken into account.

Ⅲ The wave amplitude changes with time but not with the x-direction of the global reference; in other words, the crane end-effector is assumed to be interacting with the same wave amplitude in all of its x-direction positions at a specific time instant.

The model is established using the Denavit–Hartenberg (DH) parameters in Table 1.

Table 1

DH parameters

Link	ai	αi	di	$$ \theta_i $$
1	0	$$ \frac{\pi}{2} $$	l1	q1
2	l2	0	0	q2
3	l2	0	0	q3

Hence, the four homogeneous transformation matrices from frame 0 to 3 are obtained as follows:

$$ \begin{aligned} T_1^0\left(q_1\right)=& {\left[\begin{array}{rrrr} c_1 & 0 & s_1 & 0 \\ s_1 & 0 & -c_1 & 0 \\ 0 & 1 & 0 & l_1 \\ 0 & 0 & 0 & 1 \end{array}\right], } \\ T_2^1\left(q_2\right)=& {\left[\begin{array}{rrrr} c_2 & -s_2 & 0 & l_2 c_2 \\ s_2 & c_2 & 0 & l_2 s_2 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{array}\right], } \\ T_3^2\left(q_3\right)=& {\left[\begin{array}{rrrr} c_3 & s_3 & 0 & a_3 c_3 \\ s_3 & c_3 & 0 & a_3 s_3 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{array}\right] . } \end{aligned}$$

and

$$ T_3^0(q)=\left[\begin{array}{rrrr} c_3 c_{23} & -c_1 s_{23} & s_1 & c_1\left(l_2 c_2+l_3 c_{23}\right) \\ s_1 c_{23} & -s_1 s_{23} & -c_1 & s_1\left(l_2 c_2+l_3 c_{23}\right) \\ s_{23} & c_{23} & 0 & l_1+l_2 s_2+l_3 s_{23} \\ 0 & 0 & 0 & 1 \end{array}\right] $$

The position of the end-effector, as a function of the joints angles, can be represented as:

$$ \begin{aligned} P_e = \left[ \begin{align} c_1(l_2c_2+l_3c_{23})\\ s_1(l_2c_2+l_3c_{23})\\ l_1+l_2s_2+l_3s_{23} \end{align} \right] \end{aligned} $$

where q_i, with $$ i= 1, 2, 3 $$, indicates the crane joints angles and q is the joints vector representation. Moreover, c_i and s_i denote the sine and cosine of the corresponding angle q_i, respectively.

For the action space in this environment, the effect of the hydraulic cylinders that actuate the crane links is neglected, and the crane angles are assumed to be controlled directly. Hence, the action space in this environment is described as:

$$ q_{1}=|\; 0\; \;0\; \;0 \ldots| $$

$$ q_{2}=|-0.2: 0.1: 0.2| $$

$$ q_{3}=|-8: 4:\; 8| $$

Note that q₁ is an array of zeros and only a 2D workspace is considered. Hence, in the upgraded environment, we have 25 different combinations of actions, where we take the permutations of q₂ and q₃ and set the number of elements of q₁ to 25 elements. Moreover, the dynamical model [Equation (3)] was developed using the Lagrange approach and the equation of motion, which are used to extract the torque control sequence from the optimal control sequence of joint angles.

(3) $$ M(q) \ddot{q}+C(\dot{q}, q) \dot{q}+g(q)=\tau $$

where $$ {q} $$, $$ \dot{q} $$, and $$ \ddot{q} $$ denote angle, angular velocity, and acceleration, respectively, and M($$ q $$), C($$ \dot{q} $$, $$ q $$), and g($$ q $$) denote inertial, centrifugal, and gravitational matrices, respectively.

3. RL ALGORITHMS

4. SIMULATION RESULTS

4.1. In the initial environment

4.1.1. Episode examples

Figure 3 contains acceptable landing operations, where the agent managed to land the load on the vessel with impact velocity less than 0.3 m/s.

Figure 2. Agent–environment interactions.

Figure 3. Acceptable landing episodes.

Note that, at the beginning of the episodes, the hoist velocity is fixed to 0.3 m/s; hence, the learning process takes place after the hoist reaches 2 m from the average wave amplitude. Moreover, this fixed starting action helps to avoid the delusion at the beginning of an episode, for example, going above the starting position, which would slow down the learning process. In addition, these results were obtained with the initial setting mentioned in Table 2.

4.1.2. Q-learning vs. double Q-learning

Since the algorithm is model-free, it is not possible to measure the bias that usually exists in the Q-learning algorithm. Hence, the results of the Q-learning are compared to the Double Q-learning results so the bias value can be measured. Figure 5 shows that DQL has the same sparsity $$ \alpha $$ relation as QL, but the bias of the asymptotic value between the QL and the DQL increases when the $$ \alpha $$ value is decreased.

Figure 4. Acceptable landing episodes.

Figure 5. A comparison of QL and DQL with $$ \gamma $$ = 0.97.

Moreover, it is clear that DQL does not suffer from the delusion of the values of the average return at the beginning of the learning process; on the contrary, QL does suffer from overshooting the asymptotic value, which is clear in the case of $$ \alpha = 10^{-4} $$.

Note that the episodes number in the case of $$ \alpha = 10^{-5} $$, as shown in Figure 5, had to be increased as the convergence of the DQL was delayed to the episode number 1500, and this is one of the drawbacks of the DQL, while decreasing $$ \alpha $$, the convergence to the asymptotic value delays more.

4.2. In the upgraded environment

The same Q-learning algorithm was tested in the upgraded environment, and it can be noticed that in the majority of the episodes in this case, the agent could find an accepted impact velocity at early stages of the episode time span, as shown in Figure 4.

Moreover, by using the dynamical model, the input torques corresponding to the optimal control sequence of angles of one of the episodes can be calculated, as shown in Figure 6, where it is clear that Link 2 has the highest inertia, which extends the limits of the torque domain.

Figure 6. Example of a torque optimal control sequence.

4.3. Environment comparison

Figure 7 indicates two differences between the initial environment (Ini Env) and the upgraded environment (Upg Env). First, the average return in the case of the upgraded environment is much less sparse than the initial environment. Second, the asymptotic average return values do not coincide.

Figure 7. Environments comparison.

5. DISCUSSION

The purpose of this work is to control the impact that occurs in offshore crane load landing operations in one of the most complicated environments due to the impact of several conditions. Although strong assumptions were set, two environments were established. The first considered the crane hoist as a point mass and the second was enlarged with the crane structure using a robot-like mathematical model, which was established and simulated using MATLAB and Simulink. Both environments have the sea waves model involved in the vessel's motion where the crane's load is supposed to be placed.

Optimal control sequences were generated using the Q-learning algorithm, and they managed to reach acceptable impact velocities in online learning processes. The performance of the Q-learning algorithm was tested, and we conclude the following: an intermediate reward structure is needed to overcome the effect of the agent delusion that occurs, as the reward in the usual Q-learning algorithm is delayed to the impact point; hence, less time would be required to accomplish the task.

For the same discount factor, the lower the learning rate, the less sparse the values of the average return, and the more delayed the convergence to the asymptotic value. Moreover, the asymptotic value of the average return is reduced while reducing the learning rate; although this effect is not essential in the initial environment, it is clear in more complicated environments such as the upgraded environment in our work. The change of the discount factor---for the same other hyperparameters values---is directly proportional to the asymptotic value of the average return but with no magnificent effect on the convergence time. The algorithm performance was compared to the Double Q-learning technique; hence, the bias in the Q-learning technique can be adjusted.

Reinforcement learning, in general, does not have a separate testing sample, as every learning process is online and every control sequence is unique on its own episode. In other words, we can say that the obtained control sequence has local optimality. This was verified. We tested the agents on a sample of 500 episodes and concluded that the accuracy of the agent is facing a high variance; even when we set all the algorithm hyperparameters to be fixed, the initial position of the crane where the control sequence is tested still has a significant effect in the variation of the accuracy value.

On the other hand, although the accuracy range of the best trials in the initial environment was between 71% and 80%, this range was tremendously reduced to between 0.1% and 1.5% in the upgraded environment. We attribute that to the size of action_space, as the initial environment agent has one input of 11 actions and the upgraded environment agent has three inputs and 25 actions; hence, the more complicated the action_space, the more unique the optimal control sequence and the harder it is to find a global optimal control sequence. Thus, choosing the action_space is important not only from the learning point of view but also from the feasibility in the physical domain; in this work, it is shown that avoiding setting up sudden movements and more effort to the links with high inertia are needed.

DECLARATIONS