Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Conclusion
References
All Topics
Mathematics
Probability

A New Kernel-Based Approach for NonlinearSystem Identification

Profile image of minh quangminh quang

2011, IEEE Transactions on Automatic Control

https://doi.org/10.1109/TAC.2011.2131830
visibility

…

description

17 pages

link

1 file

Abstract

We present a novel nonparametric approach for identification of nonlinear systems. Exploiting the framework of Gaussian regression, the unknown nonlinear system is seen as a realization from a Gaussian random field. Its covariance encodes the idea of "fading" memory in the predictor and consists of a mixture of Gaussian kernels parametrized by few hyperparameters describing the interactions among past inputs and outputs. The kernel structure and the unknown hyperparameters are estimated maximizing their marginal likelihood so that the user is not required to define any part of the algorithmic architecture, e.g., the regressors and the model order. Once the kernel is estimated, the nonlinear model is obtained solving a Tikhonov-type variational problem. The Hilbert space the estimator belongs to is characterized. Benchmarks problems taken from the literature show the effectiveness of the new approach, also comparing its performance with a recently proposed algorithm based on direct weight optimization and with parametric approaches with model order estimated by AIC or BIC.

Related papers

Nonlinear System Identification: A Software Concept and Examples
Pieter De Raedt

IFAC Proceedings Volumes, 1997

This paper describes a software concept for non linear system identification. The models are specified by a (pseudo-) regressor and by a mapping, both chosen by the user. Depending on the choice of regressor the model the user can obtain, e.g., NARX, NOE or state-space models. The mapping from the regressor to the model output is also chosen by the user by combining building blocks from a library of different functions expansions. Depending on the choice of the mapping the user may specify Wiener, Hammerstein, or general black-box models. The user is given a logical path to follow where she/he starts with simple models and stepwise adds more complicated nonlinearities to the mapping. All the parameters of the model can be estimated simultaneously by a gradient based method.

View PDFchevron_right
Support vector machines with composite kernels for nonlinear systems identification
Soufiane Jelali

… , 2008. IMCSIT 2008. …, 2008

In this paper, a nonlinear system identification based on support vector machines (SVM) has been addressed. A family of SVM-ARMA models is presented in order to integrate the input and the output in the reproducing kernel Hilbert space (RKHS). The performances of the different SVM-ARMA formulations for system identification are illustrated with two systems and compared with the Least Square method.

View PDFchevron_right
Identification of nonlinear non-gaussian systems by neural networks
Pavel Hering

IFAC Proceedings Volumes, 2004

Application of neural networks in identification of nonlinear non-Gaussian systems is treated. Stress is laid on a parameter estimation of the networks. They are trained by the Gaussian sum method which is a global filtering method allowing to determine conditional probability density functions of network weights. Proposed approach to estimation of network weights (parameters) based on Gaussian sum filtering method overcomes commonly used prediction error methods and it is an interesting alternative to sequential Monte Carlo methods. The considered training approach is demonstrated by an illustration example.

View PDFchevron_right
Nonlinear system identification under various prior knowledge
Zygmunt Hasiewicz, Grzegorz Mzyk

IFAC Proceedings Volumes, 2008

In the note several algorithms for nonlinear system identification are presented. The class of block-oriented dynamic nonlinear systems is considered, in particular, Hammerstein and Wiener systems are investigated. The algorithms exploit various prior knowledge -from parametric -to nonparametric. A semiparametric algorithm, which shares advantages of both approaches is proposed.

View PDFchevron_right
Implementation of Gaussian process models for nonlinear system identification
David Murray-Smith

'Proceedings 5th MATHMOD, Vienna, February 2006', Editors I. Troch, F. Breitenecker, 2006

Abstract This thesis is concerned with investigating the use of Gaussian Process (GP) models for the identification of nonlinear dynamic systems. The Gaussian Process model is a nonparametric approach to system identification where the model of the underlying system is to be identified through the application of Bayesian analysis to empirical data. The GP modelling approach has been proposed as an alternative to more conventional methods of system identification due to a number of attractive features.

View PDFchevron_right
Regularization for Nonlinear System Identification
Alessandro Chiuso

Communications and control engineering series, 2022

In this chapter we review some basic ideas for nonlinear system identification. This is a complex area with a vast and rich literature. One reason for the richness is that very many parameterizations of the unknown system have been suggested, each with various proposed estimation methods. We will first describe with some details nonparametric techniques based on Reproducing Kernel Hilbert Space theory and Gaussian regression. The focus will be on the use of regularized least squares, first equipped with the Gaussian or polynomial kernel. Then, we will describe a new kernel able to account for some features of nonlinear dynamic systems, including fading memory concepts. Regularized Volterra models will be also discussed. We will then provide a brief overview on neural and deep networks, hybrid systems identification, block-oriented models like Wiener and Hammerstein, parametric and nonparametric variable selection methods.

View PDFchevron_right
Prediction error identification of linear systems: A nonparametric Gaussian regression approach
Alessandro Chiuso

Automatica, 2011

A novel Bayesian paradigm for the identification of output error models has recently been proposed in which, in place of postulating finite-dimensional models of the system transfer function, the system impulse response is searched for within an infinite-dimensional space. In this paper, such a nonparametric approach is applied to the design of optimal predictors and discrete-time models based on prediction error minimization by interpreting the predictor impulse responses as realizations of Gaussian processes. The proposed scheme describes the predictor impulse responses as the convolution of an infinite-dimensional response with a low-dimensional parametric response that captures possible high-frequency dynamics. Overparameterization is avoided because the model involves only a few hyperparameters that are tuned via marginal likelihood maximization. Numerical experiments, with data generated by ARMAX and infinite-dimensional models, show the definite advantages of the new approach over standard parametric prediction error techniques and subspace methods both in terms of predictive capability on new data and accuracy in reconstruction of system impulse responses.

View PDFchevron_right
Nonlinear system identification via direct weight optimization
Alexander Nazin

Automatica, 2005

A general framework for estimating nonlinear functions and systems is described and analyzed in this paper. Identification of a system is seen as estimation of a predictor function. The considered predictor function estimate at a particular point is defined to be affine in the observed outputs, and the estimate is defined by the weights in this expression. For each given point, the maximal mean-square error (or an upper bound) of the function estimate over a class of possible true functions is minimized with respect to the weights, which is a convex optimization problem. This gives different types of algorithms depending on the chosen function class. It is shown how the classical linear least squares is obtained as a special case and how unknown-but-bounded disturbances can be handled.

View PDFchevron_right
Nonlinear System Identification Using Maximum Likelihood Estimation
Bindu Elias

2013

Different algorithms can be used to train the Neural Network Model for Nonlinear system identification. Here the 'Maximum Likelihood Estimation' is implemented for modeling nonlinear systems and the performance is evaluated. Maximum likelihood is a well-established procedure for statistical estimation. In this procedure first formulate a log likelihood function and then optimize it with respect to the parameter vector of the probabilistic model under consideration. Four nonlinear systems are used to validate the performance of the model. Results show that Neural Network with the algorithm of Maximum Likelihood Estimation is a good tool for system identification, when the inputs are not well defined.

View PDFchevron_right
Selective recursive kernel learning for online identification of nonlinear systems with NARX form
Ping Li

Journal of Process Control, 2010

and sharing with colleagues.

View PDFchevron_right

References (40)

  1. T. Söderström and P. Stoica, System Identification. Englewood Cliffs, NJ: Prentice-Hall, 1989.
  2. L. Ljung, System Identification, Theory for the User. Englewood Cliffs, NJ: Prentice Hall, 1997.
  3. H. Akaike, "A new look at the statistical model identification," IEEE Trans. Automat. Contr., vol. AC-19, pp. 716-723, 1974.
  4. G. Schwarz, "Estimating the dimension of a model," Ann. Statist., vol. 6, pp. 461-464, 1978.
  5. C. Hurvich and C. Tsai, "Regression and time series model selection in small samples," Biometrika, vol. 76, pp. 297-307, 1989.
  6. G. Pillonetto and G. De Nicolao, "A new kernel-based approach for linear system identification," Automatica, vol. 46, no. 1, pp. 81-93, 2010.
  7. G. Pillonetto, A. Chiuso, and G. De Nicolao, "Prediction error iden- tification of linear systems: A nonparametric Gaussian regression ap- proach," Automatica, vol. 47, no. 2, pp. 291-305, 2011.
  8. Sjoberg, Q. Zhang, L. Ljung, A. B. Delyon, P. Glo- rennec, H. Hjalmarsson, and A. Juditsky, "Nonlinear black-box mod- eling in system identification: A unified overview," Automatica, vol. 31, pp. 1691-1724, 1995.
  9. I. Lind and L. Ljung, "Regressor and structure selection in NARX models using a structured ANOVA approach," Automatica, vol. 44, pp. 383-395, 2008.
  10. I. Leontaritis and S. Billings, "Input-output parametric models for non- linear systems part II: Stochastic non-linear systems," Int. J. Contr., vol. 41, no. 2, pp. 329-344, 1985.
  11. T. Lin, B. Horne, P. Tino, and C. Giles, "Learning long-term dependen- cies in NARX recurrent neural networks," IEEE Trans. Neural Netw., vol. 7, no. 6, pp. 1329-1338, Nov. 1996.
  12. S. Shun-Feng and F. Yang, "On the dynamical modeling with neural fuzzy networks," IEEE Trans. Neural Netw., vol. 13, pp. 1548-1553, 2002.
  13. N. Draper and H. Smith, App. Regression Anal. New York: Wiley, 1981.
  14. R. Haber and H. Unbehauen, "Structure identification of nonlinear sys- tems-a survey," Automatica, vol. 26, pp. 651-677, 1990.
  15. S. Billings, A. Chen, and M. Korenberg, "Identification of MIMO non- linear systems using a forward-regression orthogonal algorithm," Int. J. Contr., vol. 49, pp. 2157-2189, 1989.
  16. W. Spinelli, L. Piroddi, and M. Lovera, "On the role of prefiltering in nonlinear system identification," IEEE Trans. Automat. Contr., vol. 50, pp. 1597-1602, 2005.
  17. J. Roll, A. Nazin, and L. Ljung, "Nonlinear system identification via direct weight optimization," Automatica, vol. 41, pp. 475-490, 2005.
  18. E. Bai and Y. Liu, "Recursive direct weight optimization in nonlinear system identification: A minimal probability approach," IEEE Trans. Automat. Contr., vol. 52, no. 7, pp. 1218-1231, 2007.
  19. C. Rasmussen and C. Williams, Gaussian Processes for Machine Learning. Cambridge, MA: The MIT Press, 2006.
  20. G. Pillonetto and B. Bell, "Bayes and empirical Bayes semi-blind de- convolution using eigenfunctions of a prior covariance," Automatica, vol. 43, no. 10, pp. 1698-1712, 2007.
  21. N. Aronszajn, "Theory of reproducing kernels," Trans. Amer. Math. Soc., vol. 68, pp. 337-404, 1950.
  22. T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning. Berlin, Germany: Springer-Verlag, 2008.
  23. E. Hannan and M. Deistler, The Statistical Theory of Linear Systems. New York: Wiley, 1988.
  24. G. Wahba, Spline Models for Observational Data. Philadelphia, PA: SIAM, 1990.
  25. H. Minh, "Reproducing kernel Hilbert spaces in learning theory," Ph.D., Brown Univ., Providence, RI, 2006.
  26. S. Saitoh, Theory of Reproducing Kernels and its Applications. New York: Longman, 1988.
  27. E. Bai, R. Tempo, and Y. Liu, "Identification of IIR nonlinear systems without prior structural information," IEEE Trans. Automat. Contr., vol. 52, no. 3, pp. 442-453, Mar. 2007.
  28. D. MacKay, "Probable networks and plausible predictions-A review of practical Bayesian methods for supervised neural networks," Net- work: Computation in Neural Systems, vol. 6, pp. 469-505, 1995.
  29. W. Gilks, S. Richardson, and D. Spiegelhalter, Markov Chain Monte Carlo in Practice. London, U.K.: Chapman and Hall, 1996.
  30. I. Steinwart, D. Hush, and C. Scovel, "An explicit description of the re- producing Kernel Hilbert space of Gaussian RBF kernels," IEEE Trans. Inf. Theory, vol. 52, no. 10, pp. 4635-4643, Oct. 2006.
  31. S. Smale and D. Zhou, "Learning theory estimates via integral oper- ators and their approximations," Constructive Approximation, vol. 26, pp. 153-172, 2007.
  32. L. Piroddi and W. Spinelli, "An identification algorithm for polynomial NARX models based on simulation error minimization," Int. J. Contr., vol. 76, pp. 1767-1781, 2003.
  33. A. Argyriou, C. A. Micchelli, and M. Pontil, "Learning convex com- binations of continuously parameterized basic kernels," in Proc. Conf. Learning Theory (COLT'05), 2005, pp. 338-352.
  34. C. A. Micchelli and M. Pontil, "Learning the kernel function via regu- larization," J. Machine Learning Res., vol. 6, pp. 1099-1125, 2005.
  35. F. Dinuzzo, Kernel Machines With Two Layers and Multiple Kernel Learning 2010 [Online]. Available: http://www-dimat.unipv.it/dinuzzo
  36. N. D. Bruijn, Asymptotic Methods in Analysis. Amsterdam, The Netherlands: North-Holland, 1961.
  37. B. Bell and G. Pillonetto, "Estimating parameters and stochastic func- tions of one variable using nonlinear measurements models," Inverse Problems, vol. 20, no. 3, pp. 627-646, 2004.
  38. B. D. O. Anderson and J. B. Moore, Optimal Filtering. Englewood Cliffs, N.J.: Prentice-Hall, 1979.
  39. M. Abramowitz and I. Stegun, Handbook of Mathematical Functions, With Formulas, Graphs, and Mathematical Tables. New York: Na- tional Bureau of Standards, 1964.
  40. Digital Library of Mathematical Function NIST [Online]. Available: http://dlmf.nist.gov/

Related papers

Kernel Identification of Non-Linear Systems with General Structure
Grzegorz Mzyk

Algorithms, 2020

In the paper we deal with the problem of non-linear dynamic system identification in the presence of random noise. The class of considered systems is relatively general, in the sense that it is not limited to block-oriented structures such as Hammerstein or Wiener models. It is shown that the proposed algorithm can be generalized for two-stage strategy. In step 1 (non-parametric) the system is approximated by multi-dimensional regression functions for a given set of excitations, treated as representative set of points in multi-dimensional space. ‘Curse of dimensionality problem’ is solved by using specific (quantized or periodic) input sequences. Next, in step 2, non-parametric estimates can be plugged into least squares criterion and support model selection and estimation of system parameters. The proposed strategy allows decomposition of the identification problem, which can be of crucial meaning from the numerical point of view. The “estimation points” in step 1 are selected to e...

View PDFchevron_right
Identification of continuous-time nonlinear systems by using a gaussian process model
Hamed Shakouri

IEEJ Transactions on Electrical and Electronic Engineering, 2008

This paper deals with a nonparametric identification of continuous-time nonlinear systems by using a Gaussian process model. Genetic algorithm is applied to train the Gaussian process prior model by minimizing the negative log marginal likelihood of the identification data. The nonlinear term of the objective system is estimated as the predictive mean function of the Gaussian process, and the confidence measure of the estimated nonlinear function is given by the predictive covariance function of the Gaussian process.

View PDFchevron_right
Kernel based partially linear models and nonlinear identification
Johan Suykens

IEEE Transactions on Automatic Control, 2000

In this note, we propose partially linear models with least squares support vector machines (LS-SVMs) for nonlinear ARX models. We illustrate how full black-box models can be improved when prior information about model structure is available. A real-life example, based on the Silverbox benchmark data, shows significant improvements in the generalization ability of the structured model with respect to the full black-box model, reflected also by a reduction in the effective number of parameters.

View PDFchevron_right
System identification based on Support Kernels Regression
Salah Zidi

2013

This paper deals with the identification of nonlinear systems using multi-kernel approach. In this context, we have improved the Support Vector Regression (SVR) method in order to identify nonlinear complex system. Our idea consists in dividing the regressor vector in several blocks, and, for each one a kernel function is used. This blockwise SVR approach is called Support Kernel Regression (SKR). Furthermore, we have proposed two methods SKR(lin-rbf) and SKR(rbf-rbf). In these two methods we have divided the regressor into two Blocks. In the SKR(lin-rbf) based approach, the linear kernel and the Gaussian kernel are used, respectively, to identify the influence of the first block and of the second block on the model. However, in the SKR(rbf-rbf) approach two gaussian kernels are used. An example is presented for qualitative comparison with the classical SVR approach based on a single kernel function. The results reveal the accuracy and the robustness of the obtained model based on o...

View PDFchevron_right
Support Vector Machines for Nonlinear Kernel ARMA System Identification
Jose Luis Alvarez

IEEE Transactions on Neural Networks, 2006

Nonlinear system identification based on Support Vector Machines (SVM) has been usually addressed by means of the standard SVM regression (SVR), which can be seen as an implicit nonlinear Auto-Regressive and Moving

View PDFchevron_right

Related topics

  • Mechanical Engineering
  • Mathematics
  • Applied Mathematics
  • Computer Science
  • Gaussian processes
  • Marginal Likelihood
  • Mixture of Gaussians
  • Nonlinear system identification
  • Gaussian Process
  • Covariance analysis
  • Electrical and Electronic Engine...

    • Cited by

    Online Aerodynamic Model Identification Using a Recursive Sequential Method for Multivariate Splines
    Qiping Chu

    Journal of Guidance, Control, and Dynamics, 2013

    Avoiding high computational loads is essential to online aerodynamic model identification algorithms, which are at the heart of any model-based adaptive flight control system. Multivariate simplex B-spline (MVSB) methods are excellent function approximation tools for modeling the nonlinear aerodynamics of high performance aircraft. However, the computational efficiency of the MVSB method must be improved in order to enable real-time onboard applications, for example in adaptive nonlinear flight control systems. In this paper, a new recursive sequential identification strategy is proposed for the MVSB method aimed at increasing its computational efficiency, thereby allowing its use in onboard system identification applications. The main contribution of this new method is a significant reduction of computational load for large scale online identification problems as compared to the existing MVSB methods. The proposed method consists of two sequential steps for each time interval, and makes use of a decomposition of the global problem domain into a number of subdomains, called modules. In the first step the B-coefficients for each module are estimated using a least squares estimator. In the second step the local B-coefficients for each module are then smoothened into a single global B-coefficient vector using a linear minimum mean

    View PDFchevron_right
    A novel Deep Neural Network architecture for non-linear system identification
    Alessandro Chiuso

    IFAC-PapersOnLine, 2021

    We present a novel Deep Neural Network (DNN) architecture for non-linear system identification. We foster generalization by constraining DNN representational power. To do so, inspired by fading memory systems, we introduce inductive bias (on the architecture) and regularization (on the loss function). This architecture allows for automatic complexity selection based solely on available data, in this way the number of hyper-parameters that must be chosen by the user is reduced. Exploiting the highly parallelizable DNN framework (based on Stochastic optimization methods) we successfully apply our method to large scale datasets.

    View PDFchevron_right
    580 California St., Suite 400
    San Francisco, CA, 94104
    © 2025 Academia. All rights reserved