Reference-shaping adaptive control by using gradient descent optimizers

This study presents a model reference adaptive control scheme based on reference-shaping approach. The proposed adaptive control structure includes two optimizer processes that perform gradient descent optimization. The first process is the control optimizer that generates appropriate control signal for tracking of the controlled system output to a reference model output. The second process is the adaptation optimizer that performs for estimation of a time-varying adaptation gain, and it contributes to improvement of control signal generation. Numerical update equations derived for adaptation gain and control signal perform gradient descent optimization in order to decrease the model mismatch errors. To reduce noise sensitivity of the system, a dead zone rule is applied to the adaptation process. Simulation examples show the performance of the proposed Reference-Shaping Adaptive Control (RSAC) method for several test scenarios. An experimental study demonstrates application of method for rotor control.


Introduction
Real world control performance strongly depends on the capability of control system to adapt itself for altering dynamics of real world applications. The real control systems should deal with change of operating conditions depending on a number of factors that lead to fluctuation of system parameters. These are ageing of system components, changing environmental conditions, hardware failures, unpredictable disturbances and noise. Since the limitations of mathematical modeling techniques, mathematical models of systems cannot foresee these effects, and therefore these effects are commonly taken into consideration as parametric uncertainty, input or output disturbance model, additive white noise, etc. in the design phase of modern control systems. Through the historical development of modern control systems, numerous robust stability and disturbance rejection control woks have addressed these topics in many aspects. However, it has been understood that if a control system is not equipped with adaption skills, unpredictability and uncertainty of the real world conditions inevitably have negative impacts on the control performance of real control systems. Although static control systems, which are not skilled to adapt themselves for changing conditions, seems to work well in simulations, they cannot exhibit the same degree of control performance in real world PLOS  applications. Even if a control system is designed perfectly robust for a certain degree of parametric uncertainty of plant model, it may still fail in real world conditions. Therefore, the focus of control engineering practice has been changing towards adaptive and intelligent control of systems and therefore adaptation skill and intelligence are becoming key assets of reliable and robust control of modern systems. Adaptive control refers to the control structures that are capable of adjusting system parameters to maintain control performance in acceptable limits. When control performance is getting worse subject to changing conditions, adaptive control systems can initiate a series of adaptation operations to restore their control performance. This process is referred to as adaptation. Therefore, the control performance evaluation is a core task of the adaptation process. In general, adaptive control systems are developed to improve real world control performance of dynamic systems. In recent years, we can see some successful implications of adaptive control techniques on the control of chaotic systems. Chaotic system models allow more relevant representation of physical world, and the control of chaotic systems model is more complicated than the control of dynamic system models, which are mainly developed for requirements of industrial control systems. Progressive developments for synchronization of complex chaotic systems have been introduced by contributions of adaptive control approaches in [1][2][3]. Recently, a stabilization method for time-dependent strict-feedback complex variable chaotic (hyperchaotic) systems with uncertain complex parameters and perturbations has been proposed by combining Lyapunov functions of complex-valued vectors and back-stepping technique [4].
In the case of a desired control performance is described by a reference model, such control systems are often called Model Reference Adaptive Control (MRAC) systems. MRAC techniques have an analogy with supervised learning approach. A mathematical reference model as a supervisory system defines the desired input-output relation for control system, and a predetermined adaptation rule tunes adjustable parameters of the control system to resemble response of the reference model.
In general, conventional MRAC systems contain two loops [20]: An inner loop includes an ordinary closed loop control structure consisting of a controller with tunable coefficients and a plant to be controlled. An outer loop is formed by the adjustment rule to tune adaptation parameters or controller parameters in order to reduce discrepancy between the reference model output and plant output, which is known as the model error [20]. Fig 1(A) depicts general structure of conventional MRAC structure.
In the current study, we present a single-loop approach, where both control and adaptation actions are carried out according to model error. Reference model is used to shape reference input signal and the approach is referred to as reference-shaping. Here, reference input of control system is not applied directly to the control system; instead, the reference input signal is first shaped by the reference model, then it is applied to the control system. Accordingly, the proposed adaptive controller follows only the output of the reference model and reduces the number of error loop to one that is model error. Control optimizer synthesizes control signal by using gradient descent optimization, which aims to reduce the model error in time. In addition to simulation results, which show response of the proposed adaptive control structure for various test scenarios, we conducted experimental study to demonstrate experimental performance of system.
Organization of paper is as follows: The next sections describe the system architecture of RSAC, present theoretical foundation and introduce derivations of gradient descent update rules. Afterward, numerical study is illustrated to evaluate response of system for various test scenarios. The further section presents some results obtained for rotors control experiments.

Methodology
System architecture and preliminaries Fig 1 shows block diagrams of conventional MRAC system and the proposed RSAC system for comparison purpose. The main difference in structures of conventional MRAC and RSAC is that the closed loop control error, which is expressed as e c (n) = r(n)−y s (n), is not used to generate control signal in RSAC structure. Instead, r(n) signal is applied to the control loop after shaped by the reference model as in Fig 1(B). Hence, the RSAC systems perform control actions only for tracking of the reference model output. In literature, input-shaping approach (reference-shaping) was applied for vibration suppression of flexible systems, a feed-forward control based on convolution of input by a sequence of pulse was utilized [27][28][29].
The model mismatch between reference model and the control system is expressed by instant model error signal (e(n)) in the form of the difference between the reference system output (y r (n)) and the plant output (y s (n)).

eðnÞ ¼ y r ðnÞ À y s ðnÞ ð1Þ
The RSAC structure includes two optimizer blocks as illustrated in Fig 1(B). One is a control optimizer to synthesize appropriate control signals for tracking of the reference model output, and the other is an adaptation optimizer that adjusts adaptation parameters according to responses of plant. Both optimizer blocks perform update rules, which work for the minimization of cost Reference-shaping adaptive control function according to gradient trends. The cost function is taken as the square of instant model error. The update equation derived for control optimizer synthesizes control signals that allow convergence of instant model error to zero. Thus, the control system output approximates to the reference model output. The update equation of adaptation optimizer works for approximation of adaptation gain to an appropriate value that characterizes instantaneous input-output relation of the plant and aids the control optimizer to generate more proper control signal so that it can follow the output of reference model. Fully overlap of system responses of a theoretical reference model and a practical control system is not always achievable because of several practical reasons such as disparity in system dynamics, model deficiencies, uncertainties, unmodeled system dynamics and nonlinearities, structural noise, etc. In long-term, these effects cause accumulation of errors and lead to prolonging adaptation efforts that may cause the instability of adaptation equations. In order to deal with such problems, an error threshold mechanism, which permits acceptable degree of model mismatches, is defined by employing a dead-zone on the model error signal. When the amplitude of model error decreases below the error threshold, the dead-zone sets zero value to the model error. The model error is used by adaptation optimizer. In this manner, operation of adaptation process is allowed to work at the moments, when the model error exceeds acceptable limits.
Gradient descent optimization is widely used for the numerically minimization of a predefined cost function. It updates parameters according to gradient trends that allow descent of the cost function. It has popular utilization in supervised learning algorithms such as back propagation training algorithm for artificial neural network [30].
Some benefits of the proposed RSAC structure can be summarized as follows: 1. RSAC does not need a plant function presume. Hence, order of plant dynamic does not increase the complexity of update rules as seen in the following section.
2. Adaptation performance of RSAC systems strongly depends on reference tracing performance of the control loop. RSAC does not need to match time responses of reference system and control systems as in conventional two-loop MRAC structures. In some cases, due to the incompatibility of system dynamics, matching of time responses of these systems may not be possible or it may require more accurate modeling of plant dynamics. In such cases, RSAC system presents advantages because RSAC system can work in the absence of an accurate modeling of the plant and therefore update rules of RSAC are obtained more simplified and fixed than update rules of MRAC. These assets are very beneficial for control practice.
Theoretical foundations for gradient descent update rules of referenceshaping adaptive control system Let us express cost function as the squared difference of outputs: An update rule for the control optimizer, which generates the control signals to minimize cost function E(n), can be written as, where parameter η c is the learning rate, and it is used to adjust the convergence speed of the gradient descent optimization method. This equation is also foundation of MIT rule. The reference model is independent of control signal, which leads to @y r @u ¼ 0. So, one can write the gradient term @E @u as, Let's express the instant input-output relation of a plant at the moment t in the form of, Eq (5) states that the value of system output at any time can be written by a time-varying factor of its current input value.
Property 1: If the gain factor k is time-varying parameter, y s (t) = k(t)u(t) represents all real valued systems.
Proof: One always finds a value of k(t) that yields output data y s (t) 2 R for an input data u(t) 2 R−{0}, such that, kðtÞ ¼ y s ðtÞ uðtÞ . Consequently, the instant input-output relation, expressed as y s (t) = k(t)u(t), becomes valid for real valued systems. This property reduces model dependence and hence the adaptive control method does not require a prior dynamic plant model presume. Instead, it estimates the current value of k(t) via gradient descent optimization. This can be a significant advantage for control practice. Thus, design and implementation complications originating from inaccurate modeling can be reduced.
In Eq (4), one can write @y s @u ¼ k. Then, update rule for the control optimizer can be obtained as, For discrete form, the finite difference is commonly used for derivative operators, and iterative update rule of control process can be written for a time sampling t = nT s as, In a similar manner, an update rule for the adaptation optimizer, which estimates instant adaptation gain k(n) to minimize E(n), can also be written according to the gradient descent as, where parameter η a is the learning rate for adaptation. The gradient term @E @k is written as, According to the Eq (5), one can write @y s @k ¼ u and the update rule for adaptation optimizer can be derived as, In discrete form, an iterative scheme of update rule for adaptation process can be written as, One of weakness of gradient descent solutions is its sensitivity to noise signal because of misleading impacts of noisy signal on gradient descent operation. The derivatives of gradient operators are sensitive to noise so that noisy spikes on signals can yield high amplitudes of derivative operators, which can results in misleading of gradient directions. On the other hand, the time response of theoretical reference model cannot exactly match the response of real systems. This is another factor that can reduce the performance of adaptation processes. In such cases, stabilization of adaptation parameters may not be possible, and lead to instability of control system as a result of overgrowing of adaptation parameters. To deal with this problem, the following dead zone rule to modify the instant model error signal is used: ( where e z is error threshold. This equation enables to govern adaptation optimizer with respect to the level of model error. It can switch on or off adaptation process operations depending on error signal magnitude. Accordingly, the proposed system performs the adaptation only for the model errors that exceeds an unacceptable model mismatch level. This modification to model error signal can be helpful to deal with implementation problems such as the misleading impacts of inherent, low-level noise signals (quantization error, system noise, etc.) on gradient descent optimization and system dynamic mismatching problems. Considering these practical concerns, the update rule for adaptation optimizer can be enhanced as, ( It can be useful to discuss stability conditions of the proposed adaptive control system, theoretically. We used Lyapunov stability theorem to investigate stability conditions of the proposed RSAC method as follows: Remark 1: For positive learning rates, η a > 0 and η c > 0, and a first order dynamic reference model in the form of T R ðsÞ ¼ a sþa , where a > 0 to ensure a stable response, output of proposed adaptive control system approximates to the output of T R (s) in time and the proposed adaptive control scheme also behaves stable.
Proof: To use Lyapunov stability to verify stability of system, let us take positive definite function E ¼ 1 2 e 2 as the Lyapunov function. Here, Lyapunov theorem suggests us that if the condition of dE dt < 0 is provided, the Lyapunov function E converges to zero. It refers that model error decreases in time and thus output of control system approximates to the output of reference model. Obviously, the control system becomes stable as long as the first order reference model T R ðsÞ ¼ a sþa is stable. Lets us show dE dt < 0: The first order reference model, given by T R ðsÞ ¼ y r ðsÞ rðsÞ ¼ a sþa , can be expressed in the time domain as, dy r dt ¼ À ay r þ ar ¼ aðr À y r Þ: By considering Eq (5), one can write the derivative of y s as, Then, considering the update rules @u @t ¼ Z c ke and @k @t ¼ Z a ue, it can be written as, Then, by using Eqs (15) and (17) in Eq (14) one obtains, For a > 0, the first-order reference model T R ðsÞ ¼ a sþa is asymptotically stable, that is, for t!1, the difference between reference input signal and reference model output goes to zero, (r−y r )!0, then it is evident that Since η a > 0 and η c > 0, the term −e 2 (η a u 2 + η c k 2 ) is always negative, so the stability condition dE dt < 0 is satisfied. The model error decreases in time and the control system output approximates to the output of the reference model. Here, we know that the reference model is always stable function because of T R ðsÞ ¼ a sþa , where a > 0. Consequently, the proposed adaptive control system becomes a stable system.
Remark 2: For strictly decreasing model error, the first-order reference model should satisfy the following condition, a < gðZ a u 2 þ Z c k 2 Þ; ð20Þ where g ¼ minðy r À y s Þ maxðrÀ y r Þ . Proof: For optimality of adaptation actions, it is very useful to find sufficient conditions that make E a monotone and strictly decreasing function. For strictly decreasing model error in time, the condition dE dt < 0 should be satisfied for all t ! 0. By using Eq (18) for dE dt , one can write, By reorganizing this inequality, one obtains the parameter a that satisfies the condition dE dt < 0 as, a < eðZ a u 2 þ Z c k 2 Þ ðr À y r Þ : In proper and causal control systems, output y r follows reference input r, and output y s follows output y r . In these cases, the sign of (r−y r ) is the same as the sign of e = (y r − y s ). Let min (e) represent the minimum value of e and max(r−y r ) represent the maximum value of (r−y r ), the following inequality is valid arithmetically. minðy r À y s ÞðZ a u 2 þ Z c k 2 Þ maxðr À y r Þ eðZ a u 2 þ Z c k 2 Þ ðr À y r Þ : ð23Þ By considering Eq (23), it is evident that parameter a can be chosen according to, The Eq (24) yields a sufficient condition that makes dE dt < 0 valid for all t ! 0. Therefore, by choosing parameter a according to Eq (20), it can be possible to have strictly decreasing response of model error in control applications. For a practical control system, the constant g ¼ minðy r À y s Þ maxðrÀ y r Þ is almost definite and hardware limitations determine bounds of (r−y r ) and (y r −y s ) parameters.

Numerical study
In this section, we performed two simulation scenarios in Matlab/Simulink environment. In the first scenario, we demonstrate effects of disturbance signal on the proposed RSAC system. In the second scenario, we investigate effects of an instant perturbation of plant function on the control performance of RSAC, and compare it with conventional MRAC.

Simulation scenario 1
In this scenario, adaptive control of a second order plant function GðsÞ ¼ 1 s 2 þsþ1 according to a reference model T R ðsÞ ¼ 0:5 sþ0:5 under an additive input disturbance is illustrated. It is noteworthy that orders of plant function G(s) and reference model T R (s) are different and this dynamic mismatch is a factor that complicates the adaptation process. Fig 2 shows the Simulink simulation model that was developed for this test scenario.
We applied a reference input signal with square waveform with the period of 150 sec and simulated the system for 30000 sec to confirm stability of the system via long-term observations. In simulations, the learning rates were set to η a = 1.5 and η c = 1. To initiate control signal generation by the recursive update rule of control process, we set a non-zero initial value for the adaptation gain k, that is, k(0) = 1. The error threshold was configured to e z = 0.09 in simulations.  To see impacts of the step disturbance, which was applied at 15000 sec in Fig 5(A), system outputs and change of the control signal are presented in Fig 5(B) and 5(C). At the onset of step disturbance, ripples increase at plant output, however control and adaptation optimizers reduce ripples. In order to illustrate contributions of error threshold e z to system performance, Fig 5(D) presents the mean square error (MSE) measured for various values of e z under a band limited white noise type disturbance insertion with power of 0.01. According to the figure, an optimal value of e z is 0.9 and further increasing of e z leads to deterioration of adaptation performance because of unnecessarily interruptions of adaptation optimizer by larger dead zones.

Simulation scenario 2
In this scenario, we investigate response of the proposed adaptive control structure for the case of an instant parametric perturbation of plant function. At the simulation time of 15000 sec, . The error threshold was configured to e z = 0.09. The learning rates and initial value of adaptation gain were set to (η a = 2, η c = 1) and k(0) = 1, respectively.  Fig 7(A) illustrates overall response of the system during 30000 sec to observe long-term performance of system. Fig 7(B) shows impacts of plant perturbation on the output of the control system. The parametric change of plant function leads to increase of overshoot and ripples at the beginning. Fig 7(C) shows the output of plant at the end of simulation. One can see that the overshoot and ripples almost disappears. These results confirm improvement of system response by adaptation efforts, after the plant perturbation. Fig 8 shows the corresponding changes in control signal, adaptation gain and model error during the adaptation process. At the first 15000 sec, system adapted itself for the initial plant function. After the plant perturbation took place, the range and amplitude of control signal were adjusted for new plant function as shown in Fig 8(A). Change in average power of controller is an indication of alterations in system properties [31]. Power estimation of controller signal decreases from 145.7 to 51.0 at 15000 sec, and this validates system perturbation. Fig 8(B) shows change of adaptation gain that enables better fitting to reference model output. Fig 9 shows the short-time average squared model error calculated by.
The figure clearly demonstrates convergence of average square error for L = 1000 and it confirms adaptation of RSAC system for the initial plant and final plant functions in time.

Reference-shaping adaptive control
It is useful to show performance of the adaptive control system for continuous multi-sinusoidal reference input signal. The plant function for this simulation is GðsÞ ¼ 1 2:3s 2 þ1:7sþ2:3 . Fig  10 reveals successful response of RSAC system for a continuous input signal. Fig 11 shows a comparison of responses of RSAC and conventional MRAC in this test scenario. The results in the figure indicate that RSAC can provide an improved reference model following performance. This improvement mainly originates from the control optimizer block that synthesizes optimized control signals to strictly decrease model error. Conventional MRAC structure with MIT rule [8,20] generates proportional control signal by using control error, and sharply changing of control error, which emerges at rising and falling edges of step reference signal, leads to high overshoots until MIT update rule responds for reduction of these overshoots. Delays in the response of MIT update rule causes the increase of the overshoot levels. In RSAC structure, sharp change in reference input cannot reduce control performance because reference input is not introduced to control optimizer. This is an important advantage of RSAC structure in practice.

Experimental study
Coaxial rotors experimental test platform

Reference-shaping adaptive control
In this experimental study, we used standard coaxial rotors and blade set of LS-222 Gyro 3.5 Channel model helicopter. For online adaptive control of coaxial rotors test platform, Arduino Mega 2560 control card was used for data acquisition. The data captured from the encoder output and control signal applied to rotors by the Arduino control card. Arduino platforms are low-cost control card solutions and suitable for the low-cost experimental digital control studies. We have designed the RSAC structure in Matlab/Simulink environment as shown in Fig 13. Arduino control card receives the angle data of shaft from the rotary encoder and it drives two electric motors of coaxial rotors by means of a Darlington power amplification circuit. Fig 14 depicts the circuit diagram of electronic parts of the experimental system.

Experimental results
In the experimental study, the reference system model was taken as T R ðsÞ ¼ 60 . The error threshold was configured to e z = 0.009. The initial value for learning rates and adaptation gain were set to (η a = 0.01, η c = 0.01) and k(0) = 0.05, respectively. A sinusoidal reference signal, r (t) = 0.3 + 0.2sin(0.1257t), is applied for the control of coaxial rotors. Fig 16 shows response of RSAC system. At the first period, shown in Fig 16(B), the output of control system cannot follow the output of reference system because adaptation to the controlled system has not occurred, yet. After the adaptation, improvement in the response of control performance manifests itself as more proper following the reference model output by control system as shown in Fig 16(C). Reference-shaping adaptive control of the adaptation process. The error signal also confirms a successful adaptation by decreasing the magnitude of ripples about the zero.
It should be noticed that the proposed method is sensitive to large dead-time system delay. The large system delays can lead to growing of output ripples. We observed that large dead time delays cause misleading of the update rules because of the update rules based on gradient descent require very recent values of error signals in order to correctly estimate gradient directions. The negative effects of large dead-time system delay on control performance should be considered in applications.

Conclusions
In summary, this study introduces a RSAC structure, which employs gradient descent optimization technique for model reference adaptive control. Theoretical foundations and practical utilization of the RSAC structure were presented in the paper.
Some important remarks can be summarized as follows: 1. The proposed RSAC structure implements two numerical gradient descent optimization processes that work in conjunction. These processes are the control optimizer for control signal generation and the adaptation optimizer for estimation of instantaneous input-output relation of plant.
2. The design and implementation of the RSAC structure are quite straightforward. Update rules are fixed and simple equations. The proposed method does not require a proper modeling of plant dynamics and it is a noteworthy asset of the proposed method for control practice. Its computational scheme uses only an estimation of current value of time-varying Reference-shaping adaptive control adaptation gain that can characterize instantaneous input-output relation of a controlled system. The conventional MRAC mainly need mathematical modeling of plant to obtain effective update rules to tune coefficients of controller functions. For this reason, complexity of update rules of conventional MRACs depends on the plant function and it can reduce applicability of conventional MRAC structures.
3. Noise sensitivity of adaptation parameters and dynamic model mismatches are the major problems, that may decrease performance of adaptive control systems in real world Reference-shaping adaptive control applications. To deal with this problem, dead-zone modification for model error signal is a common approach.
In practice, very large system delay and high-amplitude noise signal may fail the adaptation process because large dead-time delay of error signal or high-amplitude noise on model error can easily mislead update rules. These effects cause to grow the amplitude ripples around set points and cause instability of the system in time. The future works should address solutions to decrease delay and noise sensitivity of RSAC.