Continual online learning-based optimal tracking control of nonlinear strict-feedback systems: application to unmanned aerial vehicles

2.1. System description

Consider the following strict feedback system

where $$ \bar{x}_{2}(t)= $$$$ \left[x_{1}(t), x_{2}(t)\right]^{\top} \in \mathbb{R}^{2} $$ is the system state, $$ u \in \mathbb{R} $$ is the control input, and $$ f_{i}\left(.\right), g_{i}(.), i=1, 2 $$ are unknown yet Lipschitz continuous functions on $$ \Omega_{x} $$ and satisfying $$ f_i(0)=0 $$. The standard following assumptions are stated to proceed.

Assumption 1 (^[17]). The nonlinear CT system is controllable and observable. In addition, the control coefficient matrix satisfies $$ \|g_{i}(.)\| \leq g_{iM} $$, where $$ g_{iM} $$ is an unknown positive constant.

Assumption 2 (^[4]). The state vector is considered measurable. The desired trajectory $$ \xi_{r} \in \mathbb{R} $$ is bounded, and there exists a Lipschitz continuous command generator function $$ h_d\left(\xi_r(t)\right) $$ such that $$ \dot{\xi}_r(t)=h_d\left(\xi_r(t)\right) $$ and $$ h_d(0)=0 $$.

Next, the LRL control design is introduced. The goal of the LRL control scheme is to achieve satisfactory tracking and maintain the boundedness of all closed-loop system signals while minimizing the control effort and addressing the issue of catastrophic forgetting.

The design of the control system begins by implementing an optimal backstepping approach using augmented system-based actor-critic architecture and then using an online LL to mitigate catastrophic forgetting.

2.2. Optimal backstepping control

To develop optimal control using the backstepping technique, first, a new augmented system is expressed in terms of tracking error as follows. Define the tracking error as $$ e_{tr1}(t)=x_{1}(t)-\xi_{r}(t) $$. Taking the time derivative and using the value of $$ \dot{x}_{1} $$ to obtain

where $$ x_{2}(t) $$ is the virtual control.

In order to get both the feedforward and feedback part of the controller, the tracking problem is changed to a regulation problem by defining a new augmented state as $$ z_{1}=\left[\begin{array}{ll}e_{tr1}^{\top}& \xi^{\top}_{r}\end{array}\right]^{\top} $$. Then, we can write

where $$ \mathcal{F}_{s1}({z}_{1})=\bigg[\begin{array}{c} f_{1}\left({x}_{1}\right)-h_d(\xi_{r}(t)) \\ h_d(\xi_{r}(t))\end{array}\bigg] $$ and $$ \mathcal{G}_{s1}({z}_{1})=\bigg[\begin{array}{c}g_{1}({x}_{1}) \\ 0\end{array}\bigg] $$, with $$ z_{1}(0)=\left[\begin{array}{ll}e_{tr1}^{\top}(0) & \xi^{\top}_{r}(0)\end{array}\right]^{\top} $$ and $$ \alpha_{1} $$ is virtual control. From Assumption 1, it follows that $$ 0<\|\mathcal{G}_{si}(.)\| \leq \bar{G}_{i} $$, where $$ \bar{G}_{i} >0 $$ for $$ i=1, 2 $$. The design is presented in two steps.

Step 1: For the first backstepping step, let $$ \alpha_1 $$ and $$ \alpha_1^* $$ be the virtual and optimal virtual control inputs, respectively. The optimal performance index function $$ J_1^*(z_1) $$ is defined as

where $$ \Psi\left(\Omega_{z1}\right) $$ denotes the set of admissible control policies over a compact set $$ \Omega_{z1} $$, $$ {J}_{1}(z_{1})=\left(\int_{t}^{\infty} e^{-\gamma_{1}(s-t)} {h}_{1}\left(z_{1}, \alpha_{1}\right) d s\right) $$, and $$ h_1(z_1, \alpha_1) = z_1(s)^{\top}q_{1}z_1(s) + \alpha_1(z_{1})^{\top}r_{1}\alpha_1(z_1) $$ with, $$ q_1 $$ is a positive definite, $$ r_1>0 $$, $$ \gamma_{1} $$ is the discount factor, $$ \gamma_{1}>0 $$.

Remark 1Generally, addressing trajectory tracking control problems poses considerable challenges, particularly when dealing with a system characterized by nonlinear dynamics and a trajectory that evolves over time. In such instances, a prevalent approach is to employ a discounted cost function, denoted as (4), to render the cost index, $$ {J}^{*}_{1} $$, finite. The control input, consisting of both feedforward and feedback components, is obtained simultaneously by minimizing the performance function (4) along the trajectories of the augmented system (3). In addition, the performance function is not explicitly dependent on time.

By taking the time derivative on both sides of the optimal performance function (4), the tracking Bellman equation is obtained as

By noting that the first term of (5) is $$ \gamma_{1}J_{1} $$, therefore, (5) can be rewritten as

Therefore, the tracking HJB equation is generated as

where $$ H_{1}\left(z_{1}, \alpha_{1}^{*}, \frac{d J_{1}^{*}}{d z_{1}}\right) $$ is the Hamiltonian function for the first step, $$ Q_{1}=z_1(s)^{\top}q_{1}z_1(s), \bar{\alpha}_{1}=\alpha^{*}_1(z_{1})^{\top}r_{1}\alpha^{*}_1(z_1) $$. The optimal control solution $$ \alpha_1^* $$ can be obtained by solving $$ \frac{\partial H_1}{\partial \alpha_1^*}=0 $$. This equation represents the condition for finding the optimal control that minimizes the performance index and satisfies the HJB equation as

It is well known that NNs have universal function approximation abilities and can approximate a nonlinear continuous function $$ \mathcal{H}(z): \mathbb{R}^n \rightarrow \mathbb{R}^m $$ on a compact set $$ \Omega_z $$ as $$ \mathcal{H}(z) = W^{\top}\sigma(z), $$ where $$ W \in \mathbb{R}^{p \times m} $$ is the weight matrix, $$ p $$ is the number of neurons, and $$ \sigma(z) $$ is the basis function vector.

Since $$ J_{1}^{*} $$ is unknown, an NN and its derivative will be used to approximate it on a compact set as

where $$ W^{*}_{c1} $$, $$ \sigma_{c1} $$, and $$ \varepsilon_{c1} $$ are the target weights, basis function vector, and the NN reconstruction error, respectively, $$ \nabla\sigma_{c1} $$ and $$ \nabla\varepsilon_{c1} $$ are the partial derivative of $$ \sigma_{c1} $$ and $$ \varepsilon_{c1} $$, respectively, with respect to input $$ {z}_{1} $$. Substituting (8) and (9) into the HJB equation (6) to get

where optimal Hamiltonian function $$ H_{1}^{*}=H_1\left(z_1, {\alpha}_1^{*}, \frac{d {J}_1^*}{d z_1}\right), $$ and $$ \wedge_{1}=\mathcal{G}_{s1}r_{1}^{-1}\mathcal{G}^{\top}_{s1} $$. Since the target weight matrix, $$ W^{*}_{c1} $$, is unknown, an actor-critic NN and its derivative will be designed to find the solution as follows

where $$ \hat{W}_{c1} $$ is the estimate of the weight matrices for critic NN, and $$ \sigma_{c1}({z}_{1}) $$ is the basis function vector. Next, using (12), we can rewrite (11) as

where the estimated Hamiltonian function $$ \hat{H}_{1}=H_1\left(z_1, \hat{\alpha}_1, \frac{d \hat{J}_1}{d z_1}\right), \hat{\wedge}_{1}=\hat{\mathcal{G}}_{s1}r_{1}^{-1}\hat{\mathcal{G}}^{\top}_{s1} $$ and $$ \hat{\mathcal{F}}_{s1}({z}_{1}), \hat{\mathcal{G}}_{s1} $$ are the estimate of augmented internal dynamics and input dynamics to be generated in the subsequent section by using an NN identifier. From (8), $$ \alpha_{1} $$ is unknown, continuous on a compact set $$ \Omega_{z1} $$, and an NN is used to approximate it over a compact set, as shown in the subsequent steps. We can write

where $$ \mathcal{V}_{1}^{*}({z}_{1})=r^{-1}_{1}\mathcal{G}^{\top}_{s1}({z}_{1})\frac{d J_{1}^{*}\left(z_{1}\right)}{d z_{1}}. $$ where $$ \mathcal{G}_{s1}({z}_{1})\in \mathbb{R}^{2} $$, and $$ \frac{d J_{1}^{*}\left(z_{1}\right)}{d z_{1}} \in \mathbb{R}^{2} $$ which makes $$ \mathcal{V}_{1}^{*}({z}_{1}) $$ a scaler. To achieve the tracking, decompose the term $$ \mathcal{V}^{*}_{1} $$ into following two parts

where $$ \mathcal{V}_{1}^{0}=-2\beta_{1}z_{1}+\mathcal{V}_{1}^{*}, $$ where $$ \beta_{1} $$ is a $$ [\bar{\beta}_{1}, 0] $$ i.e $$ \beta_{1}\in \mathbb{R}^{1\times2} $$, $$ \bar{\beta}_{1}>0 $$, $$ z_{1} \in \mathbb{R}^{2} $$. The optimal virtual controller $$ \alpha_{1}^{*} $$ can be written as

Therefore, $$ \mathcal{V}^{0}_{1} $$ can be approximated using NN,

where $$ W^*_{a1} \in \mathbb{R}^{p \times m} $$ are the target bounded unknown weights, and $$ \varepsilon_{1} \in \mathbb{R}^m $$ denotes the function reconstruction error; $$ \sigma_{a1}({z}_{1}) $$ with $$ {z}_{1} \in \Omega_{z_{1}} $$ is the basis function vector. Hence, the virtual controller (8) can be written as

where $$ W^*_{a1} \in \mathbb{R}^{p \times m} $$ are the target bounded unknown weights, and $$ \varepsilon_{1} \in \mathbb{R}^m $$ denotes the function reconstruction error; $$ \sigma_{a1}({z}_{1}) $$ with $$ {z}_{1} \in \Omega_{z_{1}} $$ is the basis function vector. Since the target weights are unknown, the actor NN will be designed to estimate the optimal virtual control as

where $$ \hat{W}_{a1}, \sigma_{a1}({z}_{1}), {z}_{1}\in \Omega_{z1} $$ are the estimated weights and the basis function of actor NN. According to the HJB equation (11) and its approximation (13), define the HJB residual error $$ e_1(t) $$ as

since the optimal Hamiltonian value is zero. Notice that the estimated Hamiltonian, $$ \hat{H}_{1} $$, requires the estimate of unknown dynamics from the identifier. The second step is discussed next.

Remark 2The subsequent section will leverage the HJB residual error (20) to formulate the weight update laws for the critic NN. Additionally, the control input error $$ u_{e} $$, delineated in Lemma 1, will facilitate the derivation of the actor NN weight update laws.

Employing the HJB residual error and the control input error for formulating the weight update laws is pivotal as it enables efficient optimization of the weights in NN. This methodology ensures more accurate and reliable learning processes, allowing the network to better approximate the desired functions or policies, thereby enhancing the overall performance and robustness of the system.

Step$$ \boldsymbol{2} $$: This is the second step, and the actual controller $$ \hat{u} $$ will be derived. Define $$ e_{tr2}(t)=x_{2}(t)-\hat{\alpha}_{1} $$; then using (1), the error dynamics are written as

Since $$ \hat{\alpha}_{1}=-\beta_{1}{z}_{1}-{\hat{W}}^{\top}_{a1}\sigma_{z} $$, on using the values from step 1, $$ \dot{\hat{\alpha}}_{1} $$ can be calculated by using a virtual generator $$ h_{d}(\hat{\alpha}_{1}) $$. Let $$ {z}_{2}=[{e}_{tr2}^{\top}, \hat{\alpha}^{\top}_{1}]^{\top} \in \mathbb{R}^{2} $$. One can write

where $$ \mathcal{F}_{s2}({z}_{2})=\bigg[\begin{array}{cc}f_{2}(\bar{x}_{2})-h_{d}(\hat{\alpha}_{1}) \\ h_{d}(\hat{\alpha}_{1})\end{array}\bigg] $$ and $$ \mathcal{G}_{s2}({z}_{2})=\bigg[\begin{array}{cc} {g}_{2}(\bar{x}_{2})\\ 0\end{array}\bigg] $$. Letting $$ u^{*} $$ be the optimal control, the optimal integral cost function is defined as

where $$ h_{2}=z_2(s)^{\top}q_{2}z_2(s)+u^{*}(z_{2})^{\top}r_{2}u^{*}(z_2) $$ is the cost function. The user-defined penalties, $$ q_2 $$ and $$ r_2 $$, are positive definite and $$ \gamma_{2}>0 $$.

The HJB equation for step 2 is given by

where $$ H_{2}\left(z_{2}, u^{*}, \frac{d J_{2}^{*}}{d z_{2}}\right) $$ is the Hamiltonian function for the second step, $$ Q_{2}=z_2(s)^{\top}q_{2}z_2(s), \bar{u}=u^{*}(z_{2})^{\top}r_{2}u^{*}(z_2). $$ Similar to previous steps, solving $$ \left(\partial H_{2} / \partial u^{*}\right)=0 $$ yields

Since $$ J_{2}^{*} $$ is unknown, an NN will be used to approximate it on a compact set given by

Therefore, $$ \frac{dJ_{2}^{*}({z}_{2})}{d z_{2}}=\nabla\sigma^{\top}_{c2}({z}_{2}){W}^{*}_{c2}+\nabla\varepsilon^{\top}_{c2} $$, where $$ W_{c2}^{*}, \sigma_{c2}, \varepsilon_{c2} $$ are the NN weights, activation function, and the reconstruction error, respectively. Substituting (25) and (26) into the HJB equation (24) gets

where $$ H^{*}_{2}=H_2\left(z_2, {\alpha}_2^{*}, \frac{d {J}_2^*}{d z_2}\right) $$ is the optimal Hamiltonian for the second step, $$ \wedge_{2}=\mathcal{G}_{s2}r_{2}^{-1}\mathcal{G}^{\top}_{s2} $$. Since the target weight matrix, $$ W^{*}_{c2} $$, is unknown, an actor-critic NN will be designed to find the solution as follows

where $$ \hat{W}_{c2} $$ is the estimate of the weight matrices for critic NN, and $$ \sigma_{c2}({z}_{2}), {z}_{2}\in \Omega_{z2} $$ is the basis function vector. Next, we can rewrite (27) as follows

where $$ \hat{H}_{2}=H_2\left(z_2, \hat{\alpha}_2, \frac{d \hat{J}_2}{d z_2}\right) $$ is the estimated Hamiltonian, $$ \hat{\wedge}_{2}=\hat{\mathcal{G}}_{s2}r_{2}^{-1}\hat{\mathcal{G}}^{\top}_{s2} $$ and $$ \hat{\mathcal{F}}_{s2}({z}_{2}), \hat{\mathcal{G}}_{s2} $$ are the estimate of augmented internal and input dynamics to be generated in the subsequent section by using the NN identifier.

Since $$ \mathcal{G}_{s2}({z}_{2}) $$ and $$ \frac{d J_{2}^{*}\left(z_{2}\right)}{d z_{2}} $$ are unknown, continuous on a compact set $$ \Omega_{z2} $$, an NN is used to approximate them over a compact set, we can write

where $$ \mathcal{V}_{2}^{*}({z}_{2})=r^{-1}_{2}\mathcal{G}^{\top}_{s2}({z}_{2})\frac{d J_{2}^{*}\left(z_{2}\right)}{d z_{2}}. $$ To achieve tracking, decompose the term $$ \mathcal{V}^{*}_{2} $$ into following two parts

where $$ \mathcal{V}_{2}^{0}=-2\beta_{2}z_{2}+\mathcal{V}_{2}^{*}, \beta_{2}=[\bar{\beta}_{2}, 0] $$ i.e $$ \beta_{2}\in \mathbb{R}^{1\times2} $$, $$ \bar{\beta}_{2}>0 $$, $$ z_{2} \in \mathbb{R}^{2} $$. The optimal controller $$ u^{*} $$ can be written as $$ u^{*}=-\beta_{2}z_{2}-\frac{1}{2}\mathcal{V}_{2}^{0}. $$ Therefore, we can write

where $$ W^*_{a2} \in \mathbb{R}^{p \times m} $$ are the target bounded unknown weights, and $$ \varepsilon_{2} \in \mathbb{R}^m $$ denotes the function reconstruction error; $$ \sigma({z}_{2}) $$ with $$ {z}_{2} \in \Omega_{z_{2}} $$ is the basis function vector. Therefore, the actual controller (16) can be written as

Similarly, the actor NN will be designed to estimate the optimal control as

where $$ \hat{W}_{a2}, \sigma_{a2}({z}_{2}), {z}_{2}\in \Omega_{z2} $$ are the estimated weights and the basis function of actor NN. According to the HJB equation (27) and its approximation (29), define the HJB residual error $$ e_2(t) $$ as follows

since the optimal Hamiltonian for the second step is zero, notice that the estimated Hamiltonian, $$ \hat{H}_{2} $$, requires the estimate of unknown dynamics from the identifier. Similar to the two steps, the proposed method can be extended to $$ n $$th order.

Remark 3The optimal control input can be obtained by utilizing the gradient of the optimal value function (26) and an NN identifier. As a result, the critic NN outlined in (26) may be utilized to determine the actor input without the need for an additional NN. However, for the purpose of simplifying the derivation of weight update rules and subsequent stability analysis, separate NNs are employed for the actor and critic.

Next, an NN identifier will be used to approximate the unknown dynamics given by (3) and (22).

2.3. NN identifier

A single-layer NN is used to approximate both the nonlinear functions $$ \mathcal{F}_{sj} $$ and $$ \mathcal{G}_{sj} $$ where $$ j=1, 2 $$, that govern the augmented dynamics^[17]. Then, by using $$ \hat{\mathcal{G}}_{sj} $$, which is obtained from the NN identifier, the estimated control policy is applied to the nonlinear system. The approximations are represented as $$ {\mathcal{F}_{sj}}(z_{j}) = {V_{F_j}}^{\top} \sigma_{F_j}(z_{j}) + \varepsilon_{F_j}(z_{j}) $$ and $$ \mathcal{G}_{sj}(z_{j}) = V_{G_j}^{\top} \sigma_{G_j}(z_{j}) + \varepsilon_{G_j}(z_{j}) $$, where $$ V_{F_j} $$ and $$ V_{G_j} $$ are the NN's weight matrices, $$ \sigma_{F_j}(z_{j}) $$ and $$ \sigma_{G_j}(z_{j}) $$ are the activation functions, and $$ \varepsilon_{F_j}(z_{j}) $$ and $$ \varepsilon_{G_j}(z_{j}) $$ are the NN's reconstruction errors respectively. The estimated values of the internal dynamics and control coefficient matrix of the augmented system are given by $$ \hat{\mathcal{F}}_{sj}(z_{j}) =\hat {V}_{F_j}^{\top} \sigma_{F_j}(z_{j}) $$ and $$ \hat{\mathcal{G}}_{sj}(z_{j}) = \hat{V}_{G_j}^{\top} \sigma_{G_j}(z_{j}) $$, where $$ \hat{V}_{F_j} $$ and $$ \hat{V}_{G_j} $$ are the estimated NN's weight matrices. Define

where $$ \bar{\alpha}_{j}= $$$$ \left[1, \alpha_{j}\right]^{\top}, $$ is the augmented control input, $$ \alpha_{j} $$ is given by (19), (34), $$ Z_{j}=\left[\begin{array}{llll}V_{F_j}^{\top} & V_{G_j}^{\top} \end{array}\right]^{\top} \in \mathbb{R}^{2 l \times n} $$ represents the augmented NN identifier weights, and $$ \boldsymbol{\sigma}(\boldsymbol{\xi_{1}})= $$$$ \operatorname{diag}\{ \sigma_{F_j}\left(z_{j}\right), \sigma_{G_j}(z_{j})\} $$ denotes the augmented activation function for the NN identifier. The reconstruction error of the NN identifier, denoted by $$ \varepsilon_{I}(z_{j}) $$, is defined as $$ \varepsilon_{I}(z_{j})= $$$$ (\varepsilon_{F_j}(z_{j})+\varepsilon_{G_j}(z_{j}) \alpha_{j}) $$. Next, the following assumption is stated.

Assumption 3 (^[17]). The NN identifier is of single-layer, and its reconstruction error is bounded above such that $$ \left\|\varepsilon_{I}(z_{1})\right\|^{2} \leq b_{0}\|z_{1}\|^{2} $$ and $$ \|Z_{1}\| \leq Z_{m_{1}} $$.

Remark 4Because $$ \varepsilon_{I}(z_{j}) $$ depends on input $$ \alpha_{j} $$ and the system state $$ z_{j}(t) $$, it is assumed to be bounded above by the norm of the state vector unlike^[18], where $$ \varepsilon_{I}(z_{j}) $$ is bounded by a constant value.

Define the dynamics of the NN identifier as

where $$ \hat{z}_{j}(t) $$ represents the estimated augmented state vector, $$ K $$ is a user defined constant gain matrix, and $$ \hat{Z}_{j}=\left[\begin{array}{llll}\hat{V}_{F_j}^{\top} & \hat{V}_{G_j}^{\top} \end{array}\right]^{\top} $$ represents the augmented NN identifier estimated weights. The state estimation error is defined as $$ e_{i_{j}}=z_{j}-\hat{z}_{j}. $$ The weight update law for the NN identifier is given by

where $$ \alpha_{vj}>0 $$ is a tuning parameter.

Remark 5The NN identifier weights are tuned by using both augmented state estimation error and the input vector. The boundedness of the control input is needed to show the convergence of the NN identifier if proof is shown separately, whereas this assumption is relaxed when the identifier is combined with the LRL control scheme, as shown in the next section.

Remark 6Since the control input error and the HJB error used to tune the actor-critic NN weights require the system dynamics which are uncertain, the NN identifier is used to approximate the unknown dynamics of an augmented system. The estimated values from the identifier are used in the actor-critic weight update laws to tune the NN weights, as shown in the subsequent section.

2.4. Actor critic NN weight tuning

In this section, the actor-critic weight update laws are obtained using the gradient descent method to the Hamiltonian-based performance function. The following Lemma is stated.

Lemma 1Consider a system (1), transformed system (3), NN identifier weight update laws (38), the update laws for the critic NN (12), (26) and actor NN (19), (34). They can be written as

where $$ j $$ denotes the number of steps in backstepping, $$ \gamma_{cj}=\frac{\beta_{c j}\hat{\psi}_{j}}{\|\hat{\psi}_j\|^2+1} $$, $$ \beta_{c j}, \beta_{j}, \beta_{a j}>0 $$ are the learning rates, $$ S_{aj}=\sigma_{aj}\left( {z}_{j}\right) $$, $$ S_{cj}=\sigma_{cj}\left( {z}_{j}\right) $$, and $$ \hat{\psi}_j(t)=\gamma_{j}S_{cj}-\nabla\sigma_{cj}(\hat{\mathcal{F}}_{sj}+\hat{\mathcal{G}}_{sj}\hat{\alpha}_{j}) $$, where $$ \sigma_{1j}>0, \sigma_{2j>0} $$ are the design constants, $$ u_{ej} $$ is the error between estimated control and actual control and is given by $$ u_{ej}=-\beta_{j}z_{j}-\frac{1}{2}\hat{W}^{\top}_{aj}\sigma_{aj}\left( {z}_{j}\right)+\frac{1}{2}r_{j}^{-1} \hat{\mathcal{G}}^{\top}_{sj}\frac{d \hat{J}_{j}}{d z_{j}} $$, $$ e_{j} $$ is the HJB error for each step given by (20), (35), $$ \hat{\wedge}_{j}=\hat{\mathcal{G}}_{sj}r_{j}^{-1}\hat{\mathcal{G}}^{\top}_{sj}, $$ and $$ \hat{\mathcal{F}}_{sj}, \hat{\mathcal{G}}_{sj} $$ are approximated by using NN identifier.

Proof: The weight update laws for critic NN in step $$ j $$ are obtained by defining the performance function as

By using the gradient descent algorithm, the weight update law can be obtained as

On simplifying (42), we will get the weight update law for critic NN, as shown in Lemma. The weight update law for actor NN is obtained by defining the performance function as

By using the gradient descent approach, the weight update law for an actor NN is obtained as

On further solving and adding the stabilization terms, we will get the weight update law shown in Lemma 1.

Remark 7. The weight update laws are obtained using the gradient descent method to the Hamiltonian-based performance function. The weight update equations for the critic and actor have additional terms to ensure stability and facilitate convergence proof. The last term, known as the sigma modification term, relaxes the PE condition needed to ensure weight convergence. It is important to note that the right-hand side terms in the weight update equation can be measured.

The following assumption is stated next.

Assumption 4 (^[4]). It is assumed that the ideal weights exist and are bounded over a compact set by an unknown positive constant, such that $$ \left\|W_{cj}^{*}\right\| \leq \bar{W}_{M1j} $$$$ , \left\|W_{aj}^{*}\right\| \leq \bar{W}_{M2j} $$ where $$ \bar{W}_{M1j}, \bar{W}_{M2j} $$ are unknown constant values. The basis function $$ \sigma(\cdot) $$, function reconstruction error $$ \varepsilon_j(\cdot) $$, and their derivatives with respect to their arguments are assumed to be bounded over a compact set, with unknown bounds as $$ \|\sigma\|\leq\bar{\sigma}, \|\dot{\sigma}\|\leq \bar{\bar{\sigma}}, \varepsilon_j(\cdot)\leq \varepsilon_{m}, \dot{\varepsilon}_j(\cdot)\leq \bar{\varepsilon}_{m} $$.

Next, the following theorem is stated.

Theorem 1Consider the nonlinear system in strict-feedback form defined by (1). By using the augmented system (3), consider the optimal virtual control (19) and actual control terms (34), along with the identifier, actor-critic updating laws (38), (39), and (40). Further, assume that the design parameters are selected as per the conditions stated and that Assumptions 1 through 4 hold. If the system input is PE and its initial value, $$ u_0 $$, is admissible, then tracking errors, augmented state $$ z_j $$, the actual state $$ x_{j} $$ and NN weight estimation errors, $$ \tilde{{Z}}_{j}=Z_{j}-\hat{Z}_{j} $$, $$ \tilde{{W}}_{c_j}=W_{cj}-\hat{W}_{cj} $$, and $$ \tilde{{W}}_{a_j}=W_{aj}-\hat{W}_{aj} $$ are guaranteed to be bounded. This, in turn, ensures that the actual system output tracks the desired trajectory and estimated/actual control inputs are bounded close to their optimal values.

Proof: See Appendix.

Remark 8In the proposed optimal backstepping technique, the RL/ADP is employed at every step to obtain the optimal virtual and actual control inputs. We have derived the backstepping for a two-step process; however, it can be implemented up to $$ n $$ steps using a similar procedure.

Remark 9The sigma modification term does serve to alleviate the PE condition and assists in the process of forgetting; however, it does not prove effective in multitasking scenarios to minimize forgetting. Subsequently, a novel online LL strategy is presented to address the issue of catastrophic forgetting.

Next, an online regularization-based approach to LL is introduced.

2.5. Continual lifelong learning

To mitigate the issues of catastrophic forgetting^[13], a novel technique called WVA was proposed^[15]. However, WVA has only been used in an offline manner, which cannot be applied to NN-based online techniques.

In contrast, this study introduces a new online LL technique that can be integrated into an online NN-based trajectory tracking control scheme by identifying and safeguarding the most critical parameters during the optimization process. To achieve this, the proposed technique employs a performance function given by

where $$ E_{bj} $$ is the loss function for the current task $$ B $$ in step j (41), $$ \bar{\Omega}_{kj}=diag\{\frac{\Omega_{w1}}{ \Omega_{w1}+1} \ldots \frac{\Omega_{wn}}{ \Omega_{wn}+1}\} $$, $$ \Omega_{wi} $$ represents the significance of the $$ i $$-th weight, where $$ i = 1, \dots, n $$, of the NN in step $$ j $$ after learning from prior tasks, the diagonal elements of the FIM $$ \bar{\Omega}_{kj} $$, with $$ k $$ denoting the task, are estimated using HJB error since targets are unavailable in online learning, $$ \lambda_{j} $$ is the design parameter controlling the strength of the regularization, $$ \left\|\hat{W}_{cj}-\hat{W}_{cjp}^*\right\|_{ \bar{\Omega}_{kj}}=(\hat{W}_{cj} -\hat{W}^{*}_{cjp})\bar{\Omega}_{kj}(\hat{W}_{cj} -\hat{W}^{*}_{cjp})^{\top} $$, $$ \hat{W}^*_{cjp} $$ is the optimized bounded weight vector of the previous task, and $$ \hat{W}_{cj} $$ is the weight vector of the current task that needs to be optimized. The FIM for each task is calculated by defining the log-likelihood function as:

where $$ {e}_{jp} $$ represents the HJB residual error, as defined by (20) and (35) from the previous task. The term $$ p({e}_{jp}|\hat{W}_{cjp}, {z}_{jp}) $$ denotes the probability density function of the HJB error, given the input $$ z_{jp} $$ from the previous task and the weights $$ \hat{W}_{cjp} $$ at step $$ j $$ from the previous task. Calculate the Jacobian matrix as

where $$ \frac{\partial \ell(\hat{W}_{cjp}, {z}_{jp})}{\partial \hat{W}_{cjp}} $$ denotes the partial derivative of the log-likelihood function with respect to the weights from the previous task. Therefore, the estimation of FIM is obtained as

where $$ t_0 $$ denotes the task start time, $$ t_{1} $$ denotes the task end time. The elements of the FIM illustrate the extent of information concerning the HJB error that is conveyed by every weight within the network. Sure! Here's a clearer and more structured way to convey the information. For the first task, the FIM is zero. When estimating the FIM for the second task using data from the first task, the estimate remains bounded. This bounded behavior is due to the fact that the closed-loop system associated with task 1 is also bounded, as demonstrated in Theorem 1.

Subsequently, leveraging normalized gradient descent allows us to formulate an additional term in the critic weight update law. This term is derived as follows

For LL, the terms from (49) are combined with the terms from the previously defined update law that is given in Theorem 1. Next, the following theorem is stated.

Theorem 2Consider the hypothesis stated in Theorem 1, and let Assumptions 1 to 4 hold, with the LRL critic NN tuning law for j step optimal backstepping, given by

where j denotes the number of steps in backstepping, $$ \lambda_{j} $$ is the design parameter, and $$ \alpha_{j} $$ are the NN learning rates; then, $$ \tilde{W}_{cj} $$ and all closed-loop signals, including those in Theorem 1, are UUB.

Proof: See Appendix.

Remark 10From (49), when the significance of the weights increases, $$ \Omega_{w_{i}} $$ can become infinite; therefore, $$ \frac{\Omega_{wi}}{ \Omega_{w_{i}}+1} $$ will become $$ 1 $$; hence, the gradient explosion can be avoided.

Remark 11The first part of the NN weight update law in Theorem 2 is the same as in Theorem 1, whereas the second part includes regularization terms resulting from LL. Notice that the tracking and weight estimation error bounds increase due to the LRL-based control scheme because the bounding constant, $$ \bar{C}=\bar{C}_{0j}+\gamma_{penj} $$, includes additional terms that are obtained from the penalty function from LL, where $$ \bar{C}_{0j} $$ is bound when LL is absent, and $$ \gamma_{penj} $$ is because of LL in jth step. The overall stability is unaffected due to LL.

Remark 12The proposed LL method is scalable to encompass $$ n $$ tasks. A third task, $$ C $$, aims to maintain network weights in alignment with the learned weights from the preceding two tasks, implementable via single or dual penalties, given the quadratic nature of the penalties.

Remark 13The efficacy of the LL method is pronounced when Tasks 1 and 2 share informational overlap reflected in the weights, facilitating the knowledge transfer for Task 2. However, in the absence of shared weights or knowledge between non-overlapping tasks, the visible enhancement in Task 2 performance might be negligible with online LL, albeit it mitigates the catastrophic forgetting of Task 1, offering long-term benefits when reverting to Task 1.

3.1. Unmanned aerial vehicle (UAV) problem formulation and control design

Consider the UAV model depicted in Figure 1, which is characterized by two reference frames: the inertial frame $$ I=\{x_I, y_I, z_I\} $$ fixed to the earth and the body-fixed frame $$ A=\{x_A, y_A, z_A\} $$. The UAV is steered using four forces $$ F_{j=1, 2, 3, 4} $$, each produced by a respective rotor. To negate the yaw drift due to the reactive torque, rotors are arranged in two pairs, with $$ (1, 3) $$ spinning clockwise and $$ (2, 4) $$ spinning counterclockwise.

Figure 1. Quadrotor UAV.

The quadrotor dynamics can be modeled by two unique equations: (1) translational; and (2) rotational. However, these dynamic equations interrelate via the rotation matrix, rendering them as two cohesive subsystems. A holistic control strategy involves both outer and inner loop controls, corresponding to the two subsystems. The outer loop aims to execute positional control by managing the state variables of $$x, y, $$ and $$z$$ while concurrently generating command inputs for roll and pitch states in the inner loop through a correction block, dependent on a predefined yaw reference signal. The inner loop's objective is to achieve attitude control by managing the state variables of roll, pitch, and yaw.

Define $$\zeta_1(t) = [x(t), y(t), z(t)]^\top \in \mathbb{R}^3$$ as the UAV's positional state and $$\zeta_2(t) = [\dot{x}(t), \dot{y}(t), \dot{z}(t)]^\top \in \mathbb{R}^3$$ as its velocity state within the inertial frame $$I$$. A transformation relation exists: $$\zeta_2(t) = R(\eta_1) V$$. Consequently, the translational dynamic equation can be represented as

Given the underactuated nature of UAV translational dynamics, an intermediate control vector

is introduced for optimal position control derivation, and the translational dynamic can thus be reformulated as

Remark 14The relation between $$ V=\left[V_1, V_2, V_3\right]^\top \in \mathbb{R}^3 $$ and $$ u_c $$ is given by

Solving yields the control $$ u_c $$ as

Using the reference trajectory vector $$\zeta_{\text{ref}}(t) = \left[x_{\text{ref}}(t), y_{\text{ref}}(t), z_{\text{ref}}(t)\right]^\top \in \mathbb{R}^3$$, we define the tracking error variables as $$e _{\zeta_1}(t)=\zeta_1(t)-\zeta_{\text{ref}}(t)$$ and $$e _{\zeta_2}(t)=\zeta_2(t)-\alpha_{\zeta}$$, where $$\alpha_{\zeta}$$ is the virtual control.

For the coordinate transformation of rotational dynamic, the transformation relationship between rotational velocity $$\Omega$$ and the Euler angles rate of change $$\eta_2(t)=\dot{\eta}_1(t) = [\dot{\phi}(t), \dot{\theta}(t), \dot{\psi}(t)]^\top \in \mathbb{R}^3$$ is represented as:

with

Applying time derivation to both sides yields the attitude dynamic as:

The function $$f\left(\eta_1, \eta_2\right)$$ is represented as:

So, the attitude dynamic can be rephrased in strict feedback form as

Reference signals are denoted as $$\eta_{\mathrm{re}}(t) = \left[\phi_{\mathrm{re}}(t), \theta_{\mathrm{re}}(t), \psi_{\mathrm{re}}(t)\right]^\top$$. The yaw command element $$\psi_{\mathrm{re}}(t)$$ is predefined, and the roll and pitch command angles are derived as

Tracking error variables are designated as $$e_{\eta 1}(t)=\eta_1(t)-\eta_{\mathrm{re}}(t)$$ and $$e_{\eta 2}(t)=\eta_2(t)-\alpha_\eta$$, where $$\alpha_\eta \in \mathbb{R}^3$$ is the virtual control.

Therefore, using the control law (34) and the weight update laws shown in Theorems 1 and 2 for translational and attitude dynamics will drive the UAV system to track the reference trajectory, as shown in the simulations.

3.2. Simulation parameters and results

The desired position trajectory for $$ \xi_{\mathrm{re}}(t)= $$$$ [2t-20, 5, 0]^{\top} $$ and, for task 2, is $$ \xi_{\mathrm{re}}(t)= $$$$ [5 \sin (t), 5 \cos (t), t]^{\top} $$. The yaw command angle is predefined as $$ \psi_{\mathrm{re}}=\pi / 4 $$; then, the roll and pitch command signals $$ \phi_{\mathrm{re}} $$ and $$ \theta_{\mathrm{re}} $$ can be produced. The design parameter $$ \beta_1=[6, 0] $$, $$ \beta_2=[4.2, 0] $$, $$ m=5 $$, $$ \sigma_{1j}=0.75 $$ and $$ \sigma_{2j}=0.82 $$, $$ \beta_{ci}=0.6, \beta_{ai}=0.6 $$, $$ \lambda_{w}=2 $$, $$ \alpha_{w}=0.85 $$. The initial values are set as $$ z_1(0)=[1.7, 3.8, 0.41]^{\top}, z_2(0)=[2.31, 3, 0.4]^{\top} $$. In this configuration, the NN is based on a Random Vector Functional Link (RVFL) architecture. The network is structured with a single layer, where the input layer is directly connected to the output layer, bypassing traditional hidden layers. The interlayer connections are facilitated by a weight matrix $$ V $$, which is randomly initialized. This matrix $$ V $$ effectively represents the neurons in the network. Specifically, the network comprises ten neurons, each corresponding to a column in the $$ V $$ matrix. These neurons functionally link the input to the output through the RVFL mechanism. The transformation from the input to the output layer is given by $$ W^{\top}\sigma(V^{\top}X) $$, where $$ \sigma $$ denotes the activation function applied element-wise, and $$ X $$ is the input and $$ W $$ is tuned using the weight update laws in Theorems 1 and 2. Similarly, seven neurons are used for the identifier; sigmoid is used as an activation function, $$ \epsilon_{0}=0.65 $$. For attitude control, the initial values are set as $$ z_3(0)=[\pi / 4, \pi / 4, \pi / 4]^{\top}, z_4(0)=[\pi / 3, \pi / 3, \pi / 3]^{\top} $$.

We consider two task scenarios in which the reference trajectory is changed in each task as if the UAV is moving in a different path or environment. In the simulations, we have shown task 1 again to demonstrate that when the UAV returns to task 1, the LL-based control will help mitigate the catastrophic forgetting. The proposed method is able to drive the UAV to track the reference trajectory accurately, even on changing tasks. Figure 2 shows the performance of the position and attitude tracking; it indicates that using the proposed LRL method shown by the blue color, and the UAV position states can accurately follow the reference trajectory shown by the red color. The attitude tracking performance demonstrates that the UAV attitudes can also follow the reference attitudes better as compared to recent literature^[9] shown by green color. Figure 3 illustrates the tracking errors, indicating that the tracking performance of the proposed method is superior when compared with recent literature, referred to as 'r-lit'^[9]. Figure 4 illustrates the position and attitude tracking errors.

Figure 2. Actual and reference trajectories using the proposed LL-based method.

Figure 3. Tracking performance of position and attitude subsystems using LRL and recent literature (r-lit)^[9] methods.

Figure 4. Position and attitude tracking errors using proposed LRL and recent literature (r-lit)^[9] methods.

Both the system state tracking plots and positional error plots in Figure 2 and Figure 3 demonstrate the superior performance of the proposed LL method, represented by blue lines. However, the recent literature^[9] exhibits higher error, thus showing the need for LL. In contrast, the 'Lit' method, as shown in green, has a higher error rate when compared to other methods. Notably, the total average error shown in Figure 4 is low when the proposed LL method is employed over the 'Lit' method, indicating a substantial enhancement in tracking accuracy.

Figure 5 depicts torque inputs and cumulative costs where it can be seen that the cost of using the proposed method is minimal, and all the closed-loop signals are bounded. The control effort demanded by the 'Lit' method is higher in comparison to the proposed LL-based method. Figure 5 also showcases the cumulative cost. It is observed that the cost associated with Lit (shown in green color) is higher compared to the proposed LL method (represented in blue) during the tasks and as the tasks change.

Figure 5. Torque inputs and cumulative cost using proposed LRL and recent literature (r-lit)^[9] methods.

Authors' contributions

Made substantial contributions to the conception and design of the study: Ganie I, Jagannathan S

Made contributions in writing, reviewing, editing, and methodology: Ganie I, Jagannathan S

Availability of data and materials

Not applicable.

Financial support and sponsorship

The project or effort undertaken was or is sponsored by the Office of Naval Research Grant N00014-21-1-2232 and Army Research Office Cooperative Agreements W911NF-21-2-0260 and W911NF-22-2-0185.

Conflicts of interest

Both authors declared that there are no conflicts of interest.

Ethical approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Copyright

© The Author(s) 2024.

Proof of Theorem 1

Step1: Consider the Lyapunov function as follows

The time derivative of $$ L_{1} $$ is

Where $$ tr $$ denotes the trace operator. Let $$ \dot{L}_{1}=\dot{L}_{11}+\dot{L}_{12}+\dot{L}_{13}+\dot{L}_{14}+\dot{L}_{15}. $$ where $$ {L}_{11}=J_{1}(z_{1}), {L}_{12}=\frac{1}{2}e_{i1}^{\top}e_{i1}, {L}_{13}= \frac{1}{2}tr \left\{\tilde {Z}_{1}^{\top}\tilde {Z}_{1} \right\}, {L}_{14}=\frac{1}{2}tr\{\tilde{W}_{c1}\tilde{W}_{c1}^{\top}\}, {L}_{15}=\frac{1}{2}tr\{\tilde{W}_{a1}\tilde{W}_{a1}^{\top}\}. $$

Considering the first term of (51), we can write it as $$ \dot{J}_{1}(z_{1})=\nabla J_{1}\dot{z}_{1} $$.

Substituting (3), (9) in (52) gives

Substituting the value of $$ \alpha $$ from (8) in (53) leads to

Using $$ \hat{W}_{c1}=W_{c1}-\tilde{W}_{c1} $$ in (54), and simplifying, one will get

Separating the terms in (55) w.r.t actual NN weights and the terms w.r.t weight estimation error gives

Substituting $$ \hat{\mathcal{G}}_{s1}=\mathcal{G}_{s1}-\tilde{\mathcal{G}}_{s1} $$ in (56) leads to

One can further simplify (57), as follows

Using (11), we have $$ W_{c1}^{\top}\nabla\sigma_{c1} \mathcal{F}_{s1}(z_{1})-\frac{1}{2}W_{c1}^{\top}\nabla \sigma_{c1} \mathcal{G}_{s1}(z_{1})r^{-1}\mathcal{G}_{s1}(z_{1})W_{c1}\nabla\sigma_{c1}\leq-k\|z_{1}\|^{2} $$. Additionally, it is also assumed that the effect of the NN approximation error in the system dynamics is bounded above, i.e., $$ \nabla \varepsilon_{c1} \mathcal{F}_{s1}(z_{1})-\frac{1}{2}\nabla \varepsilon_{c1} \mathcal{G}_{s1}(z_{1})r^{-1}\mathcal{G}_{s1}(z_{1})W_{c1}\nabla\sigma_{c1}\leq-\bar{\epsilon}\|z_{1}\|^{2} $$. One can write (58), as follows

which can be further written as

where $$ \frac{1}{2}W_{c1}^{\top}\nabla\sigma \mathcal{G}_{s1}(z_{1})r^{-1}\mathcal{G}_{s1}\nabla\sigma \leq k_{1} $$, $$ \frac{\nabla \varepsilon_{c1}}{2}\mathcal{G}_{s1}(z_{1})r^{-1}\mathcal{G}_{s1}\nabla\sigma \leq k_{2} $$, $$ \frac{1}{2}W_{c1}^{\top}\nabla \sigma \mathcal{G}_{s1}(z_{1})r^{-1}W\nabla \sigma \leq k_{3} $$, $$ \frac{\nabla \varepsilon_{c1}}{2}\mathcal{G}_{s1}(z_{1})r^{-1}W_{c1}\nabla\sigma \leq k_{4} $$, $$ \frac{1}{2}W_{c1}^{\top}\nabla\sigma \mathcal{G}_{s1}(z_{1})r^{-1}\nabla\sigma \leq k_{5} $$, $$ \frac{\nabla \varepsilon_{c1}}{c2}\mathcal{G}_{s1}(z_{1})r^{-1}\nabla\sigma \leq k_{6} $$. Consider the second and third term of (52)

On substituting the value of $$ \dot{e}_{i1} $$ and $$ \dot{\tilde{Z}}_{1} $$ from Theorem 1, one can write RHS of (83) as follows

which can be further simplified by using the cyclic property of traces as

To simplify, we have

Consider the fourth term of (52),

Therefore, one can further simplify (65) by using Young's inequality in cross-product terms as follows

Consider the fifth term of (52) $$ \dot{L}_{15}=tr\{\tilde{W}_{a1}^{\top}\dot{W}_{a1}\} $$. On using the weight update law from Theorem 2, one can write

Using $$ \tilde{W}_{c1}=W_{c1}-\hat{W}_{c1} $$, $$ \tilde{W}_{a1}=W_{a1}-\hat{W}_{a1} $$, $$ \tilde{G}_{s1}=\mathcal{G}_{s1}-\hat{\mathcal{G}}_{s1} $$

Considering a last term, we can write

Using Young's inequality in the cross product terms, we can write

Combining (60), (64), (66) and (69) and simplifying, we have

where $$ \bar{c}_{1}=\bar{k}_{1}-c_{13}, \bar{k}_{1}>c_{13}, \bar{c}_{2}=c_{4}-\bar{k}_{4}, c_{4}>\bar{k}_{4}, \bar{c}_{3}=\alpha_{v}-k_{4}, \alpha_{v}>\bar{k}_{4}, \bar{c}_{4}=c_{12}-c_{5}, c_{12}>c_{5} $$. We can simplify it further as follows

where $$ \bar{C}_{01}=\frac{k_{2}^{2}}{c_{2}}+\frac{k_{3}^{2}}{c_{3}}+\frac{c_{0}^{2}}{c_{4}} $$. Therefore, we can write

This demonstrates that the overall closed-loop system is bounded. Since $$ \tilde{G} $$ is a function of the weight estimation error of the NN identifier, or $$ \tilde {Z} $$, and from (72), $$ \tilde{Z} $$ is bounded; as a consequence, $$ \hat {G} $$ is bounded. From (72) $$ \tilde{W} $$, identification error and system state are bounded. As a result, the control input error becomes bounded.

Step 2:

This is the final step. Consider the Lyapunov function as follows

The time derivative of $$ L_{2} $$ is

Let $$ \dot{L}_{2}=L_{1}+\dot{L}_{21}+\dot{L}_{22}+\dot{L}_{23}+\dot{L}_{24}+\dot{L}_{25}. $$ where $$ {L}_{21}=J_{2}(z_{2}), {L}_{22}=\frac{1}{2}e_{i2}^{\top}e_{i2}, {L}_{23}= \frac{1}{2}tr \left\{\tilde {Z}_{2}^{\top}\tilde {Z}_{2} \right\}, {L}_{24}=\frac{1}{2}tr\{\tilde{W}_{c2}\tilde{W}_{c2}^{\top}\}, {L}_{25}=\frac{1}{2}tr\{\tilde{W}_{a2}\tilde{W}_{a2}^{\top}\}. $$

Considering the second term of (74), substituting (3) in $$ \dot{J}_{2}(z_{2}) $$, it gives

which on further solving leads to

Using $$ \hat{W}_{c2}=W_{c2}-\tilde{W}_{c2} $$ in (76), and simplifying, one will get

Separating the terms in (77) w.r.t actual NN weights and the terms w.r.t weight estimation error gives

Substituting $$ \hat{\mathcal{G}}_{s2}=\mathcal{G}_{s2}-\tilde{\mathcal{G}}_{s2} $$ in (78) leads to

One can further simplify (79), as follows

Using (11), we have $$ W_{c2}^{\top}\nabla\sigma_{c2} \mathcal{F}_{s2}(z_{2})-\frac{1}{2}W_{c2}^{\top}\nabla \sigma_{c2} \mathcal{G}_{s2}(z_{2})r_{2}^{-1}\mathcal{G}_{s2}(z_{2})W_{c2}\nabla\sigma_{c2}\leq-k\|z_{2}\|^{2} $$. Additionally, it is also assumed that the effect of the NN approximation error in the system dynamics is bounded above, i.e., $$ \nabla \varepsilon_{c2} \mathcal{F}_{s2}(z_{2})-\frac{1}{2}\nabla \varepsilon_{c2} \mathcal{G}_{s2}(z_{2})r_{2}^{-1}\mathcal{G}_{s2}(z_{2})W_{c2}\nabla\sigma_{c2}\leq-\bar{\epsilon}\|z_{2}\|^{2} $$. One can write (80), as follows

which can be further written as

where $$ \frac{1}{2}W_{c2}^{\top}\nabla\sigma \mathcal{G}_{s2}(z_{2})r_{2}^{-1}\mathcal{G}_{s2}\nabla\sigma \leq k_{1} $$, $$ \frac{\nabla \varepsilon}{2}\mathcal{G}_{s2}(z_{2})r_{2}^{-1}\mathcal{G}_{s2}\nabla\sigma \leq k_{2} $$, $$ \frac{1}{2}W_{c2}^{\top}\nabla \sigma \mathcal{G}_{s2}(z_{2})r_{2}^{-1}W_{c2}\nabla \sigma \leq k_{3} $$, $$ \frac{\nabla \varepsilon}{2}\mathcal{G}_{s2}(z_{2})r_{2}^{-1}W_{c2}\nabla\sigma \leq k_{4} $$, $$ \frac{1}{2}W_{c2}^{\top}\nabla\sigma \mathcal{G}_{s2}(z_{2})r_{2}^{-1}\nabla\sigma \leq k_{5} $$, $$ \frac{\nabla \varepsilon}{2}\mathcal{G}_{s2}(z_{2})r_{2}^{-1}\nabla\sigma \leq k_{6} $$.

Consider the second and third term of (52)

On substituting the value of $$ \dot{e}_{i2} $$ and $$ \dot{\tilde{Z}}_{2} $$ from Theorem 1, one can write (83) as follows

which can be further simplified by using the cyclic property of traces as

To simplify, we have $$ \varepsilon^{2}_{I}(z_{2}) \leq \|z_{2}\|^{2} $$. Therefore, on using Young's inequality in $$ e_{i2}z_{2} $$, we have

Consider the fourth term of (52),

Therefore, one can further simplify (87) by using Young's inequality in cross-product terms as follows

Consider the fifth term of (52) $$ \dot{L}_{25}=tr\{\tilde{W}_{a2}^{\top}\dot{W}_{a2}\} $$. On using the weight update law from Theorem 2, one can write

Using $$ \tilde{W}_{c2}=W_{c2}-\hat{W}_{c2} $$, $$ \tilde{W}_{a2}=W_{a2}-\hat{W}_{a2} $$, $$ \tilde{\mathcal{G}}_{s2}=\mathcal{G}_{s2}-\hat{\mathcal{G}}_{s2} $$

Considering a last term, we can write

Using Young's inequality in the cross product terms, we can write

Combining (82), (86), (88) and (91) and simplifying, we have

where $$ \bar{C}_{02}=\frac{k_{11}^{2}}{c_{z1}}+\frac{k_{21}^{2}}{c_{z2}}+\frac{k_{12}^{2}}{C_{c1}}+\frac{k_{13}^{2}}{C_{a1}}+\frac{k_{23}^{2}}{C_{a2}}+\frac{k_{f1}^{2}}{C_{f1}}+\frac{k_{f2}^{2}}{C_{f2}} $$. Therefore, from (92), the bounds for $$ \|z_{k}\|, \|\tilde{W}_{ck}\| $$, $$ \|\tilde{W}_{ak}\|, \|\tilde{W}_{fk}\| $$ can be obtained as

This demonstrates that the overall closed-loop system is bounded. Since $$ \tilde{\mathcal{G}}_{sj} $$ is a function of the weight estimation error of the NN identifier, or $$ \tilde {Z}_{j} $$, and from (93), $$ \tilde{Z}_{j} $$ is bounded; as a consequence, $$ \hat {\mathcal{G}}_{sj} $$ is bounded. From (93) $$ \tilde{W}_{aj}, \tilde{W}_{cj} $$, identification error and system state are bounded. As a result, the control input error $$ u_{ej}=-\beta_{j}z_{j}-\frac{1}{2}\hat{W}_{aj}\sigma_{aj}\left( {z}_{j}\right)+\frac{1}{2}r_{j}^{-1}\mathcal{G}_{sj}\frac{d {J}_{j}}{d z_{j}} $$ becomes bounded with bound $$ e_{b} $$ which can be obtained by using the bounds from (93) and Assumption 4. Therefore, the actual control inputs are bounded close to their optimal values.

Proof of Theorem 2

The convergence of weights for Task 1 remains in alignment with Theorem 1. For Task 2, an additional term emerges in the Lyapunov proof (92) due to the regularization penalty, denoted as

where $$ {W_{A_{j}}}^{*} $$ is indicative of the optimized weights for the primary task, as verified to be bounded in Theorem 1.

Substituting $$ \hat{W}_{j}=W^{*}_{j}-\tilde{W}_{j} $$, we can rearrange equation (94) to

Employing Young's inequality to the first and third terms of (95), we get

Substituting $$ \|\tilde{W}_{j}\|\|W^{*}_{j}\|, \|\tilde{W}_{j}W_{A_{j}}^{*}\| $$ back into (95) provides

Thus, the integration of this term into the proof solely modifies the error bound to $$ \gamma_{\text{pen}_{j}} \leq \frac{\|W^{*}_{j}\|^2}{2} + \frac{\|W_{A_{j}}^{*}\|^2}{2} $$, without impacting the overarching stability of the system.

The aggregate contribution to the error bounds is calculated by adding $$ \gamma_{\text{penj}_{j}} $$ to (92), resulting in a comprehensive error bound $$ C_{j}= \bar{C}_{0j}+\gamma_{\text{penj}_{j}} $$. It is, therefore, clear from the derived equations that the error bounds experience an increase as the weights deviate from their optimal points; however, the comprehensive stability of the system remains intact.