Academia.eduAcademia.edu

Figure 16 – uploaded by Tayyaba Saeed

See full PDFdownloadDownload figure
Fig. 3.3: Highlighting the quantities involve in the chain rule for gradient of c, with respect to wit), The quantities involved in Eq. (B.54) for chaining of the derivatives due to the variable dependencies are highlighted in Fig. 3.3. The derivate of c, for an instance p with respect to the weight Wi is given by,

Figure 3 3: Highlighting the quantities involve in the chain rule for gradient of c, with respect to wit), The quantities involved in Eq. (B.54) for chaining of the derivatives due to the variable dependencies are highlighted in Fig. 3.3. The derivate of c, for an instance p with respect to the weight Wi is given by,

Related Figures (62)

Fig. 1.1: Machine Learning versus Deep Learning: Binary classification of appropriately pre- =~ NNs are composed of multiple interconnected processing nodes or units, called neurons, that work ygether to learn from data and do ML tasks. For supervised learning, this is achieved by training the etwork using a set of labelled data and adjusting the connections between neurons to minimize the ‘ror between the predicted output and the target output. The neurons are arranged in multiple \terconnected layers, which hierarchically represent the patterns in the data. These layers can be seen ; a sequence of non-linear transformations that map the input data to a higher-level representation, hich captures or extracts increasingly complex features and patterns from raw data without requiring xplicit feature engineering. Other ML algorithms, such as decision trees or support vector machines, ften require hand-crafted features that are specifically designed for the task at hand. In contrast, neural etworks can learn these features directly from the data, allowing them to generalize better and perform ell on a wide range of tasks. Neural networks can also handle large amounts of data and are capable f learning from unstructured and high-dimensional data such as images, audio, and text. This makes em well-suited for tasks such as image classification, speech recognition, and natural language rocessing. Thus, ANNs give rise to the field of deep learning. The distinguishing peculiarity of neural etworks from conventional machine learning algorithms is their ability to learn directly from the raw ata. Machine learning techniques consist of a series of steps: pre-processing, feature extraction, feature slection, and prediction (classification or whatever task is at hand). The model’s accuracy depends on \e quality of these handcrafted features, and expert knowledge plays a crucial role in generalization. m the other hand, deep learning using neural networks can automatically learn/extract complex and ierarchical features through multiple layers of computation. This eliminates the need for manual ature engineering, as the model learns representations directly from the raw data. This concept is epicted in Fig. 1.1.
Fig. 1.1: Machine Learning versus Deep Learning: Binary classification of appropriately pre- =~ NNs are composed of multiple interconnected processing nodes or units, called neurons, that work ygether to learn from data and do ML tasks. For supervised learning, this is achieved by training the etwork using a set of labelled data and adjusting the connections between neurons to minimize the ‘ror between the predicted output and the target output. The neurons are arranged in multiple \terconnected layers, which hierarchically represent the patterns in the data. These layers can be seen ; a sequence of non-linear transformations that map the input data to a higher-level representation, hich captures or extracts increasingly complex features and patterns from raw data without requiring xplicit feature engineering. Other ML algorithms, such as decision trees or support vector machines, ften require hand-crafted features that are specifically designed for the task at hand. In contrast, neural etworks can learn these features directly from the data, allowing them to generalize better and perform ell on a wide range of tasks. Neural networks can also handle large amounts of data and are capable f learning from unstructured and high-dimensional data such as images, audio, and text. This makes em well-suited for tasks such as image classification, speech recognition, and natural language rocessing. Thus, ANNs give rise to the field of deep learning. The distinguishing peculiarity of neural etworks from conventional machine learning algorithms is their ability to learn directly from the raw ata. Machine learning techniques consist of a series of steps: pre-processing, feature extraction, feature slection, and prediction (classification or whatever task is at hand). The model’s accuracy depends on \e quality of these handcrafted features, and expert knowledge plays a crucial role in generalization. m the other hand, deep learning using neural networks can automatically learn/extract complex and ierarchical features through multiple layers of computation. This eliminates the need for manual ature engineering, as the model learns representations directly from the raw data. This concept is epicted in Fig. 1.1.
Fig. 2.2: A schematics diagram of a perceptron.
Fig. 2.2: A schematics diagram of a perceptron.
Fig. 2.3: A schematics diagram of a multilayer perceptron or FNN Step 2 (Adding Biases):
Fig. 2.3: A schematics diagram of a multilayer perceptron or FNN Step 2 (Adding Biases):
Several definitions of the activation functions are available in the literature, each with its unique properties and advantages. Examples include the Threshold, Sigmoid, Tanh, ReLU, Leaky ReLU. Softmax functions, etc. Some of the well-known activation functions are listed in Table 2.1. The choice of activation function can have a significant impact on the performance of the model. The choice depends on the problem and data at hand and the characteristics of the data. Therefore, experimenting with different activation functions is often a good practice to see which works best. Research contributions proposed several other activation functions and demonstrated improved performance on some benchmarks. Agostinelli et al. (2015) proposed adaptive piecewise linear activation functions. Hendrycks and Gimpel (2020) proposed a high performance activation function, the Gaussian Error Linear Unit (GELU) as a variant to ReLU and ELU.
Several definitions of the activation functions are available in the literature, each with its unique properties and advantages. Examples include the Threshold, Sigmoid, Tanh, ReLU, Leaky ReLU. Softmax functions, etc. Some of the well-known activation functions are listed in Table 2.1. The choice of activation function can have a significant impact on the performance of the model. The choice depends on the problem and data at hand and the characteristics of the data. Therefore, experimenting with different activation functions is often a good practice to see which works best. Research contributions proposed several other activation functions and demonstrated improved performance on some benchmarks. Agostinelli et al. (2015) proposed adaptive piecewise linear activation functions. Hendrycks and Gimpel (2020) proposed a high performance activation function, the Gaussian Error Linear Unit (GELU) as a variant to ReLU and ELU.
Adding the biases into it, the pre-activation values for layer | are expressed in vector form as:
Adding the biases into it, the pre-activation values for layer | are expressed in vector form as:
Table 2.2: Some commonly used loss/cost functions for neural networks.
Table 2.2: Some commonly used loss/cost functions for neural networks.
Therefore, for the Quadratic cost function, Eq. (B.9) becomes If the Sigmoid function is used for the activation, then
Therefore, for the Quadratic cost function, Eq. (B.9) becomes If the Sigmoid function is used for the activation, then
The quantities involved in Eq. (B.53) for chaining of the derivatives due to the variable dependencies are highlighted in Fig. 3.2.
The quantities involved in Eq. (B.53) for chaining of the derivatives due to the variable dependencies are highlighted in Fig. 3.2.
Fig. 3.2: Highlighting the quantities involve in the chain rule for gradient of c, with respect to wy. Fig. 3.1: A network with two hidden layers.
Fig. 3.2: Highlighting the quantities involve in the chain rule for gradient of c, with respect to wy. Fig. 3.1: A network with two hidden layers.
Fig. 3.4: Highlighting the quantities involve in the chain rule for gradient of c, with respect to Wit The quantities involved in Eq. (B.55) for chaining of the derivatives due to the variable dependencies are highlighted in Fig. 3.4.
Fig. 3.4: Highlighting the quantities involve in the chain rule for gradient of c, with respect to Wit The quantities involved in Eq. (B.55) for chaining of the derivatives due to the variable dependencies are highlighted in Fig. 3.4.
Here, @ is a learning rate. a is a hyperparameter. It serves as the step size to control how much the weights and biases are to be changed. In this way, all the parameters are updated using the respective averaged gradients. Once the gradients of the parameters (weights and biases) are ready, as in Eqs. (G.1,G.2), then according to the Gradient Descent strategy, the parameters are updated as follows. The averaged gradient of C with respect to a specific parameter serves as the respective component of the vector of advancement step sizes in the Descent direction. This step is used to update the relevant parameter.
Here, @ is a learning rate. a is a hyperparameter. It serves as the step size to control how much the weights and biases are to be changed. In this way, all the parameters are updated using the respective averaged gradients. Once the gradients of the parameters (weights and biases) are ready, as in Eqs. (G.1,G.2), then according to the Gradient Descent strategy, the parameters are updated as follows. The averaged gradient of C with respect to a specific parameter serves as the respective component of the vector of advancement step sizes in the Descent direction. This step is used to update the relevant parameter.
Fig. 4.1: A comparison of error drop pattern among the three variants of the Gradient Descent method: (a) the Batch GD method, (b) the SGD method, and (c) the Mini-Batch GD method. reach the minimum. A comparison of the error drop pattern for different values of the learning rate is shown in Fig. 4.2.
Fig. 4.1: A comparison of error drop pattern among the three variants of the Gradient Descent method: (a) the Batch GD method, (b) the SGD method, and (c) the Mini-Batch GD method. reach the minimum. A comparison of the error drop pattern for different values of the learning rate is shown in Fig. 4.2.
Fig. 4.2: A comparison of different values of the learning rate
Fig. 4.2: A comparison of different values of the learning rate
Fig. 5.1: A flow chart to illustrate the neural network model training process Fig. 5.1 depicts a work flow of the generic process of training a neural network model.
Fig. 5.1: A flow chart to illustrate the neural network model training process Fig. 5.1 depicts a work flow of the generic process of training a neural network model.
After standardization, the updated values are given in Table 6.2. 6.3. Selection of the Network Structure
After standardization, the updated values are given in Table 6.2. 6.3. Selection of the Network Structure
Table 6.1: A small dataset of an example problem: Titanic Survivors The dataset shown in Table 6.1 is for illustrative purposes. It is taken from a famous dataset for classifying a Titanic passenger as survived: Yes or No (kaggle-titanic, 2017). The dataset has three input (independent) features/attributes: Class: Passenger class (1 = First, 2 = Second, 3 = Third), Age (in years), and Fare. The target variable Survived is a Boolean variable indicating whether the passenger survived (1 = Yes, 0 = No). The dataset has instances p, where p = 1,2,-::,6. 6.2. Standardization of the Dataset
Table 6.1: A small dataset of an example problem: Titanic Survivors The dataset shown in Table 6.1 is for illustrative purposes. It is taken from a famous dataset for classifying a Titanic passenger as survived: Yes or No (kaggle-titanic, 2017). The dataset has three input (independent) features/attributes: Class: Passenger class (1 = First, 2 = Second, 3 = Third), Age (in years), and Fare. The target variable Survived is a Boolean variable indicating whether the passenger survived (1 = Yes, 0 = No). The dataset has instances p, where p = 1,2,-::,6. 6.2. Standardization of the Dataset
Fig. 6.1: The Structure of the neural network 4. Initializing Weights and Biases For the hidden layer, | = 1:
Fig. 6.1: The Structure of the neural network 4. Initializing Weights and Biases For the hidden layer, | = 1:
Calculating gradients with respect to the weights using Eq. (E£.3): For the Sigmoid activation function together with the Cross-Entropy cost function, we have for j = 1,2 and i = 1,2,3: Learning Phase: Computing the Gradients through Backpropagation
Calculating gradients with respect to the weights using Eq. (E£.3): For the Sigmoid activation function together with the Cross-Entropy cost function, we have for j = 1,2 and i = 1,2,3: Learning Phase: Computing the Gradients through Backpropagation
For the Sigmoid activation function together with the Cross-Entropy cost function, we have for k = 1,2,3 andi = 1,2,3: Gradient of the cost function with respect to the weights and biases used in the layer I = 1:
For the Sigmoid activation function together with the Cross-Entropy cost function, we have for k = 1,2,3 andi = 1,2,3: Gradient of the cost function with respect to the weights and biases used in the layer I = 1:
For k = 1,i = 2:
For k = 1,i = 2:
For k = 2,i = 2:
For k = 2,i = 2:
Using the Gradient Descent method, with the learning rate a = 0.5, the parameters (weights and biases) are updated as follows. 8. Learning Phase: Updating the Parameters (Weights and Biases)
Using the Gradient Descent method, with the learning rate a = 0.5, the parameters (weights and biases) are updated as follows. 8. Learning Phase: Updating the Parameters (Weights and Biases)
Fig. 7.1: An example of the structure of M matrix, for the input layer 1 =, nz = 2, and n, = 3 Y: ann, X m matrix of true values/targets/labels, corresponding to the instances in M, such that its pth column vector has the form:
Fig. 7.1: An example of the structure of M matrix, for the input layer 1 =, nz = 2, and n, = 3 Y: ann, X m matrix of true values/targets/labels, corresponding to the instances in M, such that its pth column vector has the form:
For each layer 1, where | = 1,2,3,-:-, L, a different WI matrix is required.
For each layer 1, where | = 1,2,3,-:-, L, a different WI matrix is required.
Fig. 7.2: An example of the structure of W" matrix, for layer 1 = 2, nz = 2, andn, = 3.
Fig. 7.2: An example of the structure of W" matrix, for layer 1 = 2, nz = 2, andn, = 3.
Fig. 7.3: An example of the structure of Bl matrix, for layers 1 = 1 and 2. For each layer 1, where ! = 1,2,3,-:-, L, a different Bll vector is required.
Fig. 7.3: An example of the structure of Bl matrix, for layers 1 = 1 and 2. For each layer 1, where ! = 1,2,3,-:-, L, a different Bll vector is required.
ZI; an n, X m matrix of pre-activation neurons (linear neurons) such that its pth column vector has the form: IU@) is the vector of pre-activation neurons in layer l/ of the network for instance p. It has n, omponents because there are n; neurons in layer /. Thus, the matrix Z ] — yl pl-11 4 Bl has the orm (horizontal stacking of m column vectors for vectorized implementation) (also see Fig. 7.4):
ZI; an n, X m matrix of pre-activation neurons (linear neurons) such that its pth column vector has the form: IU@) is the vector of pre-activation neurons in layer l/ of the network for instance p. It has n, omponents because there are n; neurons in layer /. Thus, the matrix Z ] — yl pl-11 4 Bl has the orm (horizontal stacking of m column vectors for vectorized implementation) (also see Fig. 7.4):
For each layer 1, where / = 1,2,3,::-,L, a different Z [1] matrix is required.
For each layer 1, where / = 1,2,3,::-,L, a different Z [1] matrix is required.
For each layer 1, where | = 1,2,3,-:-, L, a different VY!) matrix is required. Moreover, ylol = m. Fig. 7.4: An example of the structure of Z [1] matrix, for layer 1 = 1.
For each layer 1, where | = 1,2,3,-:-, L, a different VY!) matrix is required. Moreover, ylol = m. Fig. 7.4: An example of the structure of Z [1] matrix, for layer 1 = 1.
Note that for 1 = L, Yl! represents the final output neurons (predicted values) in the output layer L. Thus, for m instances,
Note that for 1 = L, Yl! represents the final output neurons (predicted values) in the output layer L. Thus, for m instances,
C: a scalar value as the cost function, which is the average of errors or loss function values present ir the vector C,,. C can be expressed as:
C: a scalar value as the cost function, which is the average of errors or loss function values present ir the vector C,,. C can be expressed as:
Its pth component, dZ (1) is a column vector of the gradients of C with respect to the pre-activation neurons in layer / of the network for instance p. It has n; components because there are n; neurons in layer 1. As dZ [] is formed for layer |. Therefore, for each 1 = 1,2,3,-:-,L, a different dZ [1] matrix is required. For the output layer, L, for instance p, dZ H1@) is given by
Its pth component, dZ (1) is a column vector of the gradients of C with respect to the pre-activation neurons in layer / of the network for instance p. It has n; components because there are n; neurons in layer 1. As dZ [] is formed for layer |. Therefore, for each 1 = 1,2,3,-:-,L, a different dZ [1] matrix is required. For the output layer, L, for instance p, dZ H1@) is given by
For the layers, ! = L — 1, L — 2,:--,1, for instance p, AZ) is given by d¥'41: an n;, X m matrix of gradients of the cost function C with respect to the post-activation neurons (non-linear neurons) at the output layer, Z, such that its pth column vector has the form:
For the layers, ! = L — 1, L — 2,:--,1, for instance p, AZ) is given by d¥'41: an n;, X m matrix of gradients of the cost function C with respect to the post-activation neurons (non-linear neurons) at the output layer, Z, such that its pth column vector has the form:
he matrix dY'! has the form (horizontal stacking of m column vectors for vectorized implementation) dw"): an n, X (n)_1) matrix of gradients of the cost function C with respect to each of the weights such that its jth row is the transpose of the column vector of the form:
he matrix dY'! has the form (horizontal stacking of m column vectors for vectorized implementation) dw"): an n, X (n)_1) matrix of gradients of the cost function C with respect to each of the weights such that its jth row is the transpose of the column vector of the form:
dW," is the vector of gradients of C with respect to each of nj_, weights that were used for generating
dW," is the vector of gradients of C with respect to each of nj_, weights that were used for generating
For each layer 1, where / = 1,2,3,::-,L, a different dB [4] vector is required. 7.2. The Procedure
For each layer 1, where / = 1,2,3,::-,L, a different dB [4] vector is required. 7.2. The Procedure
If (C < tol), then terminate the loop over k. End for k 7.2.2. Vectorized Implementation in Matrix-Form
If (C < tol), then terminate the loop over k. End for k 7.2.2. Vectorized Implementation in Matrix-Form
Fig. 8.1: A basic approach train-test splitting and model building and testing
Fig. 8.1: A basic approach train-test splitting and model building and testing
The validation error is used to ensure the validity of the model. It checks whether our model is too simple to learn underlying patterns or too complex to learn everything in the training set. The learning algorithm that performs poorly on the training set by producing large training error (with reference to a baseline/human-level performance) is said to underfit. This indicates high bias, which comes from developing a simpler model using fewer parameters and lesser degrees of freedom. The learning algorithm that performs well on the training data (with reference to a baseline/human-level performance), but not well on the validation set is said to overfit the training data. This indicates high variance, which comes from developing a complex model using too many parameters and higher degrees of freedom. Thus, the bias-variance tradeoff is to be taken care of for a balanced learning. That
The validation error is used to ensure the validity of the model. It checks whether our model is too simple to learn underlying patterns or too complex to learn everything in the training set. The learning algorithm that performs poorly on the training set by producing large training error (with reference to a baseline/human-level performance) is said to underfit. This indicates high bias, which comes from developing a simpler model using fewer parameters and lesser degrees of freedom. The learning algorithm that performs well on the training data (with reference to a baseline/human-level performance), but not well on the validation set is said to overfit the training data. This indicates high variance, which comes from developing a complex model using too many parameters and higher degrees of freedom. Thus, the bias-variance tradeoff is to be taken care of for a balanced learning. That
Fig. 8.3: Identifying high bias and high variance using the training and validation errors. said, it is vital to strike a balance between model complexity and performance on the unseen data. Fig. 8.3 shows the performance of three different training example and explains how to detect overfitting and underfitting. The holdout validation technique usually works reasonably well. However, there are some flaws with it. The results obtained from holdout validation can be highly dependent on the specific way in which the data is split into training and validation sets. This can lead to variability in the performance of the model and make it difficult to compare different hyperparameters. Evaluating the model performance with a small validation set may to lead to select suboptimal choices of hyperparameters. A large validation set makes the reduced training set smaller. In that case, a smaller training set will be used fot training the model. The hyperparameters tuned on a smaller set will then be used for model training on the original dataset. This issue is addressed by the k-fold cross validation technique.
Fig. 8.3: Identifying high bias and high variance using the training and validation errors. said, it is vital to strike a balance between model complexity and performance on the unseen data. Fig. 8.3 shows the performance of three different training example and explains how to detect overfitting and underfitting. The holdout validation technique usually works reasonably well. However, there are some flaws with it. The results obtained from holdout validation can be highly dependent on the specific way in which the data is split into training and validation sets. This can lead to variability in the performance of the model and make it difficult to compare different hyperparameters. Evaluating the model performance with a small validation set may to lead to select suboptimal choices of hyperparameters. A large validation set makes the reduced training set smaller. In that case, a smaller training set will be used fot training the model. The hyperparameters tuned on a smaller set will then be used for model training on the original dataset. This issue is addressed by the k-fold cross validation technique.
Fig. 8.1: Using Learning curves to detect a point for the Early Stopping. The Early Stopping is kind of regularization that is used for iterative learning algorithms. In this technique, the performance of the model is monitored during the validation process using the development set. While training a model, the graphs of the training error and the validation error can be plotted after each epoch. These graphs are called as the learning curves. The trends of these graphs can help detect overfitting and underfitting. The epochs of the training are stopped when the validation error stops decreasing over a number of epochs. The model parameters, only those for which the validation error gets reduced, are saved. When the iterative process is terminated the lasted saved effective model parameters are used. Unlike the Batch Gradient Descent method, noticing the attainment of minimum level of validation error is not easy in case of the Stochastic and Mini-batch Gradient Descent method, because the error curves are not smooth. For such cases, once the error somewhat rises again over the minimum level for several epochs, the iterative process is stopped. Fig. 8.1 depicts this technique.
Fig. 8.1: Using Learning curves to detect a point for the Early Stopping. The Early Stopping is kind of regularization that is used for iterative learning algorithms. In this technique, the performance of the model is monitored during the validation process using the development set. While training a model, the graphs of the training error and the validation error can be plotted after each epoch. These graphs are called as the learning curves. The trends of these graphs can help detect overfitting and underfitting. The epochs of the training are stopped when the validation error stops decreasing over a number of epochs. The model parameters, only those for which the validation error gets reduced, are saved. When the iterative process is terminated the lasted saved effective model parameters are used. Unlike the Batch Gradient Descent method, noticing the attainment of minimum level of validation error is not easy in case of the Stochastic and Mini-batch Gradient Descent method, because the error curves are not smooth. For such cases, once the error somewhat rises again over the minimum level for several epochs, the iterative process is stopped. Fig. 8.1 depicts this technique.
Fig. 8.2: Using Learning curves to detect the overfitting.
Fig. 8.2: Using Learning curves to detect the overfitting.
2. Normalize the activations by subtracting the mean and dividing by the standard deviation: Here, ¢ is a small constant added for numerical stability.
2. Normalize the activations by subtracting the mean and dividing by the standard deviation: Here, ¢ is a small constant added for numerical stability.
Thus, the matrix Vl! has the form (vertical stacking of n,; column vectors for vectorized implementation): In the vectorized form, the Eqs. (G.5) and (G. 6) can be written as,
Thus, the matrix Vl! has the form (vertical stacking of n,; column vectors for vectorized implementation): In the vectorized form, the Eqs. (G.5) and (G. 6) can be written as,
Related topics:
Computer Science
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved