Academia.eduAcademia.edu

Figure 3 – uploaded by Elena Tovbis

See full PDFdownloadDownload figure
Figure 3. Level curves and paths of the optimization algorithms for function f5. The path of three considered algorithms is shown in Figure 3. Here, theoretical results of the influence of the orthogonality degree of matrix learning vectors on the convergence rate of the method are confirmed. The BFGS_V method performs forced orthogonalization, which improves the result of the BFGS method. The trajectories of the methods are listed in Tables Al—A3 of Appendix A (the trajectory of the DFP method is shown partially).

Figure 3 Level curves and paths of the optimization algorithms for function f5. The path of three considered algorithms is shown in Figure 3. Here, theoretical results of the influence of the orthogonality degree of matrix learning vectors on the convergence rate of the method are confirmed. The BFGS_V method performs forced orthogonalization, which improves the result of the BFGS method. The trajectories of the methods are listed in Tables Al—A3 of Appendix A (the trajectory of the DFP method is shown partially).

Related Figures (9)

k k Figure 1. Step of process (13) on hyperplane <z", c> = y, along the direction z*. Proof. Property 2 is justified by the direct implementation of the function in (15) which is the minimizing step along the direction zk which is presented in Figure 1. Property 1 follows from the fact that movement to the point cK+] is carried out along the normal to the hyperplane <z* ,C> = Yx, that is, along the shortest path (Figure 1). Movement to other points on the hyperplane, for example to point A, satisfy only the condition in (14). Let us denote the residual as * = ck — c*. By subtracting c* from both sides of (13) and naking transformations, we obtain the following learning algorithm in the form of residuals:
k k Figure 1. Step of process (13) on hyperplane <z", c> = y, along the direction z*. Proof. Property 2 is justified by the direct implementation of the function in (15) which is the minimizing step along the direction zk which is presented in Figure 1. Property 1 follows from the fact that movement to the point cK+] is carried out along the normal to the hyperplane <z* ,C> = Yx, that is, along the shortest path (Figure 1). Movement to other points on the hyperplane, for example to point A, satisfy only the condition in (14). Let us denote the residual as * = ck — c*. By subtracting c* from both sides of (13) and naking transformations, we obtain the following learning algorithm in the form of residuals:
Figure 2. Qualitative behavior of the spectrum of matrix Hk eigenvalues for cases of scaling (124) fot various values of K.
Figure 2. Qualitative behavior of the spectrum of matrix Hk eigenvalues for cases of scaling (124) fot various values of K.
Table 1. Results of minimization with normalization of matrix (124) at K = 1 and n = 1000. Table 2. Results of minimization with normalization of matrix (124) at K = 10,000 and n = 1000. For results marked with an asterisk, K = 100.
Table 1. Results of minimization with normalization of matrix (124) at K = 1 and n = 1000. Table 2. Results of minimization with normalization of matrix (124) at K = 10,000 and n = 1000. For results marked with an asterisk, K = 100.
The initial normalization of the metric matrices, as follows from the results of Tables 1 and 2, significantly improves the convergence of QNMs. The situation corresponds to the case in Figure 2 for K > 1. Large eigenvalues in the unexplored part of the subspace make it easy to find new conjugate directions and efficiently train metric matrices with almost orthogonal training vectors.
The initial normalization of the metric matrices, as follows from the results of Tables 1 and 2, significantly improves the convergence of QNMs. The situation corresponds to the case in Figure 2 for K > 1. Large eigenvalues in the unexplored part of the subspace make it easy to find new conjugate directions and efficiently train metric matrices with almost orthogonal training vectors.
Table 3. Results of minimization with normalization of matrix (124) at K = 0.000001 and n = 2.
Table 3. Results of minimization with normalization of matrix (124) at K = 0.000001 and n = 2.
Table A1. Trajectory of the BFGS_V method moving.
Table A1. Trajectory of the BFGS_V method moving.
Table A2. Trajectory of the BFGS method moving. Table A3. Trajectory of the DFP method moving.
Table A2. Trajectory of the BFGS method moving. Table A3. Trajectory of the DFP method moving.
Table A3. Cont.
Table A3. Cont.
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved