Academia.eduAcademia.edu

Figure 13 – uploaded by Muzhaffar M. Ibrahim

See full PDFdownloadDownload figure
Quadruple-Tank System Figure 3.8 illustrates the Quadruple-tank Agent block, which provided consists of two main blocks: the Environment and the Quadratic Tank Model. The Environment block includes components for defining Observations vector, calculating rewards, and stopping the simulation based on certain conditions. The Quadratic Tank Model represents the

Figure 3 Quadruple-Tank System Figure 3.8 illustrates the Quadruple-tank Agent block, which provided consists of two main blocks: the Environment and the Quadratic Tank Model. The Environment block includes components for defining Observations vector, calculating rewards, and stopping the simulation based on certain conditions. The Quadratic Tank Model represents the

Related Figures (27)

Common Learning systems can be divided into three main categories based on the type of feedback they use: supervised, unsupervised, and reinforcement learning. Supervisec learning invo ves training a model to map inputs to outputs using labeled data. Unsuper- vised learning, on the other hand, identifies hidden patterns in data without any labelec inputs. Rein environment, This approac orcement learning (RL) is distinct in that it learns by interacting with the using feedback from these interactions to improve future performance [21] h is particularly relevant in practical applications, making RL a vibrant arez of research with uses in fields such as computer science, artificial intelligence, operation: research, robotics, and control systems.
Common Learning systems can be divided into three main categories based on the type of feedback they use: supervised, unsupervised, and reinforcement learning. Supervisec learning invo ves training a model to map inputs to outputs using labeled data. Unsuper- vised learning, on the other hand, identifies hidden patterns in data without any labelec inputs. Rein environment, This approac orcement learning (RL) is distinct in that it learns by interacting with the using feedback from these interactions to improve future performance [21] h is particularly relevant in practical applications, making RL a vibrant arez of research with uses in fields such as computer science, artificial intelligence, operation: research, robotics, and control systems.
A Markov Decision Process (MDP) is formally defined by the following components [54]: referred to as the tail sub-problem, and then systematically addressing the preceding stages
A Markov Decision Process (MDP) is formally defined by the following components [54]: referred to as the tail sub-problem, and then systematically addressing the preceding stages
2.2.2 Reinforcement Learning Framework until the entire problem is comprehensively solved. This methodical approach ensures a structured and efficient resolution of complex optimization challenges.
2.2.2 Reinforcement Learning Framework until the entire problem is comprehensively solved. This methodical approach ensures a structured and efficient resolution of complex optimization challenges.
Policy Iteration Methods: Unlike Value Iteration (VI) methods, Policy Iteratio1 (PI) methods necessitate an initially admissible policy, which must be stabilizing and hav a finite cost V(-). The P and policy update steps. optimality equation and value is determined by so methods then proceed by alternating between policy evaluatio1 Equations and act as consistency checks derived from the Bellmai are resolved at each time step (k). When a policy is given, it: ving ] Equation 2.5 until it converges, which represents the policy evaluation phase. In systems with a finite state space, this phase is akin to solving a linea system of equations. Subsequently, an enhanced policy is calculated using Equation 2.6 marking the policy update phase.
Policy Iteration Methods: Unlike Value Iteration (VI) methods, Policy Iteratio1 (PI) methods necessitate an initially admissible policy, which must be stabilizing and hav a finite cost V(-). The P and policy update steps. optimality equation and value is determined by so methods then proceed by alternating between policy evaluatio1 Equations and act as consistency checks derived from the Bellmai are resolved at each time step (k). When a policy is given, it: ving ] Equation 2.5 until it converges, which represents the policy evaluation phase. In systems with a finite state space, this phase is akin to solving a linea system of equations. Subsequently, an enhanced policy is calculated using Equation 2.6 marking the policy update phase.
With this method, the algorithm computes a strictly improved policy and convergence to the optimal policy and value under the assumption that the system is controllable, as shown in [24],|48]. Algorithm 2 gives the basic template for the PI method. In general, the VI methods are less computationally demanding than the PI methods as they require only a one-step recursion in the value update step as opposed to solving the system of equations in the policy evaluation step. This characteristic would be shown in the results of the research.
With this method, the algorithm computes a strictly improved policy and convergence to the optimal policy and value under the assumption that the system is controllable, as shown in [24],|48]. Algorithm 2 gives the basic template for the PI method. In general, the VI methods are less computationally demanding than the PI methods as they require only a one-step recursion in the value update step as opposed to solving the system of equations in the policy evaluation step. This characteristic would be shown in the results of the research.
This section outlines the experimental framework, developed using MATLAB and Simulink for the design and evaluation of Reinforcement Learning (RL) and Proportional-Integral- Derivative (PID) controllers as the benchmark. The core components of this setup are depicted in Figure 3.1 which draws heavily upon the foundational research presented in the 2021 paper, ” Reinforcement Learning for Control of Valves” authored by Rajesh Siraskar. The kev components of our exnerimental setup include: This section outlines the experimental framework, developed using MATLAB and Simulink,
This section outlines the experimental framework, developed using MATLAB and Simulink for the design and evaluation of Reinforcement Learning (RL) and Proportional-Integral- Derivative (PID) controllers as the benchmark. The core components of this setup are depicted in Figure 3.1 which draws heavily upon the foundational research presented in the 2021 paper, ” Reinforcement Learning for Control of Valves” authored by Rajesh Siraskar. The kev components of our exnerimental setup include: This section outlines the experimental framework, developed using MATLAB and Simulink,
Figure 3.2: System of measuring water level with constant outlet flow On the other hand, the quadruple-tank system is more complex and consists of three main components: the tanks (21, 2%2,73,24), the electrical pumps (u1,u2), and the control valves (71,772). Each tank in this system has its own dynamics, and the interaction between the tanks adds to the complexity of the control strategy. The electrical pumps are respon- sible for supplying the necessary flow to the tanks, while the control valves regulate the distribution of this flow among the tanks. The primary objective of the control strategy in the quadruple-tank system is to maintain the voltage level u supplied to the input pumps. This voltage must be constrained within the range [0, co] volts to ensure safe and efficient operation. The control strategy must also consider the interactions between the tanks and adjust the pump voltages and valve positions to maintain the desired liquid levels in all four tanks.
Figure 3.2: System of measuring water level with constant outlet flow On the other hand, the quadruple-tank system is more complex and consists of three main components: the tanks (21, 2%2,73,24), the electrical pumps (u1,u2), and the control valves (71,772). Each tank in this system has its own dynamics, and the interaction between the tanks adds to the complexity of the control strategy. The electrical pumps are respon- sible for supplying the necessary flow to the tanks, while the control valves regulate the distribution of this flow among the tanks. The primary objective of the control strategy in the quadruple-tank system is to maintain the voltage level u supplied to the input pumps. This voltage must be constrained within the range [0, co] volts to ensure safe and efficient operation. The control strategy must also consider the interactions between the tanks and adjust the pump voltages and valve positions to maintain the desired liquid levels in all four tanks.
Figure 3.3: Mathematical Modeling of Single Water Tank System
Figure 3.3: Mathematical Modeling of Single Water Tank System
Table 3.1: Single-tank process parameters in the tanks, denoted by H to the desired set points, hg. from the tank denoted as a. The primary control objective is to regulate the water levels
Table 3.1: Single-tank process parameters in the tanks, denoted by H to the desired set points, hg. from the tank denoted as a. The primary control objective is to regulate the water levels
Figure 3.5: Quadruple-Tank System Modelling in Simulink Table 3.2: Quadruple tank process parameters
Figure 3.5: Quadruple-Tank System Modelling in Simulink Table 3.2: Quadruple tank process parameters
PID is using a function of feedback error in time-domain
PID is using a function of feedback error in time-domain
Figure 3.9: Reward Strategy for Single-Tank System provides feedback to the agent about the success of its actions in achieving the desired Equation assigns a reward of -100 if the absolute error is within a threshold, otherwise, The reward strategy is crucial as it guides the agent’s learning process. The reward function
Figure 3.9: Reward Strategy for Single-Tank System provides feedback to the agent about the success of its actions in achieving the desired Equation assigns a reward of -100 if the absolute error is within a threshold, otherwise, The reward strategy is crucial as it guides the agent’s learning process. The reward function
The methodology seeks to develop a reinforcement learning (RL) control architecture with two distinct levels of modeling complexity: a single-tank system and a quadruple-tank system. For each model, three different algorithms are employed to identify the optimal control configuration. The unique backbone and mathematical framework of each algorithm result in distinct processes for achieving the optimal policy. These processes vary in several aspects, including the duration of training, the stability of control, the performance in continuous cases, and the balance between exploration and exploitation. The single-tank system serves as a simpler model to test the basic principles of RL control, while the quadruple-tank system introduces additional complexity and challenges, providing a more rigorous test of the algorithms’ capabilities. By comparing the outcomes across these two levels of complexity, the study aims to gain insights into the scalability and robustness of the RL control architecture. Furthermore, the differences in algorithmic approaches highlight the importance of selecting appropriate methods based on the specific requirements and constraints of the control system being modeled. The methodology seeks to develop a reinforcement learning (RL) control architecture with
The methodology seeks to develop a reinforcement learning (RL) control architecture with two distinct levels of modeling complexity: a single-tank system and a quadruple-tank system. For each model, three different algorithms are employed to identify the optimal control configuration. The unique backbone and mathematical framework of each algorithm result in distinct processes for achieving the optimal policy. These processes vary in several aspects, including the duration of training, the stability of control, the performance in continuous cases, and the balance between exploration and exploitation. The single-tank system serves as a simpler model to test the basic principles of RL control, while the quadruple-tank system introduces additional complexity and challenges, providing a more rigorous test of the algorithms’ capabilities. By comparing the outcomes across these two levels of complexity, the study aims to gain insights into the scalability and robustness of the RL control architecture. Furthermore, the differences in algorithmic approaches highlight the importance of selecting appropriate methods based on the specific requirements and constraints of the control system being modeled. The methodology seeks to develop a reinforcement learning (RL) control architecture with
the concept of episodical traininig to improve the RL performance will also be provided. This chapter provides a comprehensive analysis of control systems, starting with the single tank system where PID (Proportional-Integral-Derivative) and DDPG (Deep Deterministic Policy Gradient) are benchmarked. It then compares the performance of SAC (Soft Actor- Critic), DDPG, and PPO (Proximal Policy Optimization) in this context, highlighting SAC’s, DDPG’s, and PPO’s control performance. The discussion then shifts to the more complex quadruple-tank system, where the same algorithms are evaluated. In this chapter, the concept of episodical traininig to improve the RL performance will also be provided. In Figure 4.1, the performance of Adaptive Dynamic Programming (ADP) in establish- ing control law over an extended period is depicted. The figure illustrates the progressive enhancement of control capabilities from the 100th episode to the 500th episode. Each line in the graph represents a distinct phase of improvement, highlighting the incremental advancements in both stability and robustness. This progression underscores the efficacy of ADP in refining control mechanisms, as evidenced by the continuous and measurable In Figure 4.1, the performance of Adaptive Dynamic Programming (ADP) in establish-
the concept of episodical traininig to improve the RL performance will also be provided. This chapter provides a comprehensive analysis of control systems, starting with the single tank system where PID (Proportional-Integral-Derivative) and DDPG (Deep Deterministic Policy Gradient) are benchmarked. It then compares the performance of SAC (Soft Actor- Critic), DDPG, and PPO (Proximal Policy Optimization) in this context, highlighting SAC’s, DDPG’s, and PPO’s control performance. The discussion then shifts to the more complex quadruple-tank system, where the same algorithms are evaluated. In this chapter, the concept of episodical traininig to improve the RL performance will also be provided. In Figure 4.1, the performance of Adaptive Dynamic Programming (ADP) in establish- ing control law over an extended period is depicted. The figure illustrates the progressive enhancement of control capabilities from the 100th episode to the 500th episode. Each line in the graph represents a distinct phase of improvement, highlighting the incremental advancements in both stability and robustness. This progression underscores the efficacy of ADP in refining control mechanisms, as evidenced by the continuous and measurable In Figure 4.1, the performance of Adaptive Dynamic Programming (ADP) in establish-
Figure 4.2: Algorithm Comparison for Single- Tank System between SAC, PPO, and DDPG
Figure 4.2: Algorithm Comparison for Single- Tank System between SAC, PPO, and DDPG
Table 4.1: Performance Comparison between SAC, PPO, DDPG, and PIL he trade-offs between stability and robustness among the different RL controllers.
Table 4.1: Performance Comparison between SAC, PPO, DDPG, and PIL he trade-offs between stability and robustness among the different RL controllers.
Figure 4.3: Algorithm Comparison for Single-Tank System between RL and PID The robustness of SAC highlights its effectiveness in handling variability and disturbances within the system.
Figure 4.3: Algorithm Comparison for Single-Tank System between RL and PID The robustness of SAC highlights its effectiveness in handling variability and disturbances within the system.
In the case with higher complexity, which could be defined as Multi Input and Multi output (MIMO) to control four different tank in Quadruple-Tank System, to assess the effectiveness of the proposed control approach for level control of a quadruple tank process, we conducted numerical simulations. The parameters of the process are detailed in Table 3.2. As shown in Table 1, the system is minimum phase since 7, + 72 < 2. The desired steady-state water tank levels were set to hg, = 25, hag = 16, hag = 20, and hgg = 21 respectively. Initial conditions for the water tank levels were hi(0) = ho(0) = h3(0) = ha(0) = 0.
In the case with higher complexity, which could be defined as Multi Input and Multi output (MIMO) to control four different tank in Quadruple-Tank System, to assess the effectiveness of the proposed control approach for level control of a quadruple tank process, we conducted numerical simulations. The parameters of the process are detailed in Table 3.2. As shown in Table 1, the system is minimum phase since 7, + 72 < 2. The desired steady-state water tank levels were set to hg, = 25, hag = 16, hag = 20, and hgg = 21 respectively. Initial conditions for the water tank levels were hi(0) = ho(0) = h3(0) = ha(0) = 0.
Table 4.2: Performance Comparison of each Tank for SAC Agent for handling continuous action spaces [54]. As depicted in Figure A.5, PPO algorithms often exhibit a tendency to diverge ove time. This phenomenon can be attributed to several factors, as explored by Timothy et al in their 2015 study [62]. The research findings suggest that PPO’s strengths lie primarily i1 handling discrete action spaces with simple model architecture such as SISO. This preferenc can be attributed to the prevalence of PPO implementations in domains involving discret« game manipulation, such as Atari and board games [44]. The discrete nature of thes environments aligns well with PPO’s underlying mechanisms, facilitating effective learnings and control
Table 4.2: Performance Comparison of each Tank for SAC Agent for handling continuous action spaces [54]. As depicted in Figure A.5, PPO algorithms often exhibit a tendency to diverge ove time. This phenomenon can be attributed to several factors, as explored by Timothy et al in their 2015 study [62]. The research findings suggest that PPO’s strengths lie primarily i1 handling discrete action spaces with simple model architecture such as SISO. This preferenc can be attributed to the prevalence of PPO implementations in domains involving discret« game manipulation, such as Atari and board games [44]. The discrete nature of thes environments aligns well with PPO’s underlying mechanisms, facilitating effective learnings and control
Figure A.1: Training Monitor of DDPG Algorithm in Single-tank system
Figure A.1: Training Monitor of DDPG Algorithm in Single-tank system
Figure A.2: Training Monitor of SAC Algorithm in Single-tank system
Figure A.2: Training Monitor of SAC Algorithm in Single-tank system
Figure A.3: Training Monitor of PPO Algorithm in Single-tank system
Figure A.3: Training Monitor of PPO Algorithm in Single-tank system
Figure A.4: Training Monitor of SAC Algorithm in Quadruple-tank system
Figure A.4: Training Monitor of SAC Algorithm in Quadruple-tank system
Figure A.5: Training Monitor of PPO Algorithm in Quadruple-tank system
Figure A.5: Training Monitor of PPO Algorithm in Quadruple-tank system
Related topics:
Reinforcement Learning
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved