Academia.eduAcademia.edu

Outline

Title
Abstract
Introduction
Overview of the Utl Challenge
Overview of Proceedings
Case Studies
Summary
References

ICML-2011 Workshop on Unsupervised and Transfer learning

Profile image of Vincent LemaireVincent Lemaire

Abstract

We organized a data mining challenge in \unsupervised and transfer learning" (the UTL challenge) followed by a workshop of the same name at the ICML 2011 conference in Bellevue, Washington. This introduction presents the highlights of the outstanding con- tributions that were made, which are regrouped in this issue of JMLR W&CP. Novel methodologies emerged to capitalize on large volumes of unlabeled data from tasks related (but di�erent) from a target task, including a method to learn data kernels (similarity measures) and new deep architectures for feature learning.

Related papers

A Survey on Deep Transfer Learning
Savvas Karatsiolis

Artificial Neural Networks and Machine Learning – ICANN 2018, 2018

As a new classification platform, deep learning has recently received increasing attention from researchers and has been successfully applied to many domains. In some domains, like bioinformatics and robotics, it is very difficult to construct a large-scale well-annotated dataset due to the expense of data acquisition and costly annotation, which limits its development. Transfer learning relaxes the hypothesis that the training data must be independent and identically distributed (i.i.d.) with the test data, which motivates us to use transfer learning to solve the problem of insufficient training data. This survey focuses on reviewing the current researches of transfer learning by using deep neural network and its applications. We defined deep transfer learning, category and review the recent research works based on the techniques used in deep transfer learning.

View PDFchevron_right
Impact of Deep Learning on Transfer Learning : A Review
Adnan Mohsin Abdulazeez

2021

Transfer learning and deep learning approaches have been utilised in several real-world applications and hierarchical systems for pattern recognition and classification tasks. However, in few of the real-world machine learning situations, this presumption does not sustain since there are instances where training data is costly or tough to gather and there is continually a necessity to produce high-performance learners competent with more easily attained data from diverse fields. The objective of this review is to determine more abstract qualities at the greater levels of the representation, by utilising deep learning to detach the variables in the outcomes, formally outline transfer learning, provide information on present solutions, and appraise applications employed in diverse facets of transfer learning and deep learning. This can be attained by rigorous literature exploration and discussion on all presently accessible techniques and prospective research studies on transfer learn...

View PDFchevron_right
A Review of Deep Transfer Learning and Recent Advancements
Hamid Arabnia

ArXiv, 2022

A successful deep learning model is dependent on extensive training data and processing power and time (known as training costs). There exist many tasks without enough number of labeled data to train a deep learning model. Further, the demand is rising for running deep learning models on edge devices with limited processing capacity and training time. Deep transfer learning (DTL) methods are the answer to tackle such limitations, e.g., fine-tuning a pre-trained model on a massive semi-related dataset proved to be a simple and effective method for many problems. DTLs handle limited target data concerns as well as drastically reduce the training costs. In this paper, the definition and taxonomy of deep transfer learning is reviewed. Then we focus on the subcategory of network-based DTLs since it is the most common types of DTLs that have been applied to various applications in the last decade.

View PDFchevron_right
Guest Editorial: Special Issue on New Advances in Deep-Transfer Learning
Yusuke Nojima

IEEE Transactions on Emerging Topics in Computational Intelligence, 2019

View PDFchevron_right
Unsupervised feature learning and deep learning: A review and new perspectives
Aaron Courville

arXiv preprint arXiv:1206.5538, 2012

Abstract: The success of machine learning algorithms generally depends on data representation, and we hypothesize that this is because different representations can entangle and hide more or less the different explanatory factors of variation behind the data. Although domain knowledge can be used to help design representations, learning can also be used, and the quest for AI is motivating the design of more powerful representation-learning algorithms. This paper reviews recent work in the area of unsupervised feature ...

View PDFchevron_right
A Concise Review of Transfer Learning
Hamid Arabnia

2020 International Conference on Computational Science and Computational Intelligence (CSCI), 2020

The availability of abundant labeled data in recent years led the researchers to introduce a methodology called transfer learning, which utilizes existing data in situations where there are difficulties in collecting new annotated data. Transfer learning aims to boost the performance of a target learner by applying another related source data. In contrast to the traditional machine learning and data mining techniques, which assume that the training and testing data lie from the same feature space and distribution, transfer learning can handle situations where there is a discrepancy between domains and distributions. These characteristics give the model the potential to utilize the available related source data and extend the underlying knowledge to the target task achieving better performance. This survey paper aims to give a concise review of traditional and current transfer learning settings, existing challenges, and related approaches.

View PDFchevron_right
Functional Knowledge Transfer with Self-supervised Representation Learning
Muskaan Chopra

2023 IEEE International Conference on Image Processing (ICIP)

This work investigates the unexplored usability of self-supervised representation learning in the direction of functional knowledge transfer. In this work, functional knowledge transfer is achieved by joint optimization of self-supervised learning pseudo task and supervised learning task, improving supervised learning task performance. Recent progress in selfsupervised learning uses a large volume of data, which becomes a constraint for its applications on small-scale datasets. This work shares a simple yet effective joint training framework that reinforces human-supervised task learning by learning selfsupervised representations just-in-time and vice versa. Experiments on three public datasets from different visual domains, Intel Image, CIFAR, and APTOS, reveal a consistent track of performance improvements on classification tasks during joint optimization. Qualitative analysis also supports the robustness of learnt representations. Source code and trained models are available on GitHub 1 .

View PDFchevron_right
Transfer Learning in Deep Neural Networks
Prithu Banerjee

Transfer learning (TL), have been becoming extremely popular due to its appalling property of being able to work at a different domain, where it's not trained. Often training at such domain is very costly due to various reasons such as lack of sufficient data etc. Recently Deep Neural Network (DNN) based models have been widely used in the context of TL. These networks as clear transferable layers representing common latent features, hence intuitive to transfer. In this paper we survey various DNN architectures, different training methods adopted to train those and their performance in several application domains.

View PDFchevron_right
Transfer learning: a friendly introduction
Asmaul Hosna

Journal of Big Data

Infinite numbers of real-world applications use Machine Learning (ML) techniques to develop potentially the best data available for the users. Transfer learning (TL), one of the categories under ML, has received much attention from the research communities in the past few years. Traditional ML algorithms perform under the assumption that a model uses limited data distribution to train and test samples. These conventional methods predict target tasks undemanding and are applied to small data distribution. However, this issue conceivably is resolved using TL. TL is acknowledged for its connectivity among the additional testing and training samples resulting in faster output with efficient results. This paper contributes to the domain and scope of TL, citing situational use based on their periods and a few of its applications. The paper provides an in-depth focus on the techniques; Inductive TL, Transductive TL, Unsupervised TL, which consists of sample selection, and domain adaptation...

View PDFchevron_right
Analysis of the IJCNN 2007 agnostic learning vs. prior knowledge challenge
Gavin Cawley

Neural Networks, 2008

We organized a challenge for IJCNN 2007 to assess the added value of prior domain knowledge in machine learning. Most commercial data mining programs accept data pre-formatted in the form of a table, with each example being encoded as a linear feature vector. Is it worth spending time incorporating domain knowledge in feature construction or algorithm design or can off-the-shelf programs working directly on simple low-level features do better than skilled data analysts? To answer these questions, we formatted five datasets using two data representations. The participants to the "prior knowledge" track used the raw data, with full knowledge of the meaning of the data representation. Conversely, the participants to the "agnostic learning" track used a pre-formatted data table, with no knowledge of the identity of the features. The results indicate that black-box methods using relatively unsophisticated features work quite well and rapidly approach the best attainable performances. The winners on the prior knowledge track used feature extraction strategies yielding a large number of low-level features. Incorporating prior knowledge in the form of generic coding/smoothing methods to exploit regularities in data is beneficial, but incorporating actual domain knowledge in feature construction is very time consuming and seldom leads to significant improvements. The AL vs. PK challenge web site remains open for post-challenge submissions: http://www.agnostic.inf.ethz.ch/ .

View PDFchevron_right

References (65)

  1. A. Acharya et al. Transfer learning with cluster ensembles. In ICML 2011 Unsupervised and Transfer Learning Workshop. JMLR W&CP, this volume, 2012.
  2. F. Aiolli. Transfer learning by kernel meta-learning. In ICML 2011 Unsupervised and Transfer Learning Workshop. JMLR W&CP, this volume, 2012.
  3. G. M. Allenby and P. E. Rossi. Marketing models of consumer heterogeneity. Journal of Econometrics, 89:57-78, 1999.
  4. G. M. Allenby and P. E. Rossi. Learning multiple tasks with kernel methods. Journal of Machine Learning Research, 6:615-637, 2005.
  5. A. Arnold, R. Nallapati, and W. W. Cohen. A comparative study of methods for transduc- tive transfer learning. In In ICDM Workshop on Mining and Management of Biological Data, 2007.
  6. N. Arora, G. M. Allenby, and J. Ginter. A hierarchical bayes model of primary and secondary demand. Marketing Science, 1998.
  7. B. Bakker and T. Heskes. Task clustering and gating for bayesian multi-task learning. Journal of Machine Learning Research, 4:83-99, 2003.
  8. P. Baldi. Autoencoders, unsupervised learning, and deep architectures. In ICML 2011 Unsupervised and Transfer Learning Workshop. JMLR W&CP, this volume, 2012.
  9. J. Baxter. Theoretical models of learning to learn. Learning to Learn, pages 71-94, 1997.
  10. S. Ben-David and R. Schuller. Exploiting task relatedness for multiple task learning. In Proceedings of Computational Learning Theory (COLT), pages 185-192, 2003.
  11. Y. Bengio. Learning deep architectures for AI. Foundations and Trends in Machine Learn- ing, 2(1):1-127, 2009. doi: 10.1561/2200000006. Also published as a book. Now Publish- ers, 2009.
  12. Y. Bengio. Deep learning of representations for unsupervised and transfer learning. In ICML 2011 Unsupervised and Transfer Learning Workshop. JMLR W&CP, this volume, 2012.
  13. Y. Bengio, J.-F. Paiement, P. Vincent, O. Delalleau, Nicolas Le Roux, and Marie Ouimet. Out-of-sample extensions for lle, isomap, mds, eigenmaps, and spectral clustering. In NIPS03, 2003.
  14. A. Blum and T. Mitchell. Combining labeled and unlabeled data with co-training. In COLT' 98 Proceedings of the eleventh annual conference on Computational learning theory, pages 92-100. Morgan Kaufmann Publishers, 1998.
  15. L. Breiman and J. H. Friedman. Predicting multivariate responses in multiple linear regres- sion. Royal Statistical Society Series B, 1998.
  16. J. Bromley, I. Guyon, Y. LeCun, E. Säckinger, and R. Shah. Signature verification using a "siamese" time delay neural network. In In NIPS Proc, 1994.
  17. J. Buhmann et al. Information theoretic model selection for pattern analysis. In ICML 2011 Unsupervised and Transfer Learning Workshop. JMLR W&CP, this volume, 2012.
  18. R. Caruana. Multitask learning. Machine Learning, 28(1):41-75, 1997.
  19. S.-C. Chin and W. Nick Street. Divide and transfer: an exploration of segmented trans- fer to detect wikipedia vandalism. In ICML 2011 Unsupervised and Transfer Learning Workshop. JMLR W&CP, this volume, 2012.
  20. D. Erhan, Y. Bengio, A. Courville, P.-A. Manzagol, P. Vincent, and S. Bengio. Why does unsupervised pre-training help deep learning? JMLR, 11:625-660, 2010.
  21. Z. Ghahramani. Unsupervised Learning, volume 3176, pages 72-112. Springer-Verlag, Berlin, 2004.
  22. A. Globerson and N. Tishby. Sufficient dimensionality reduction. J. Mach. Learn. Res., 3:1307-1331, March 2003. ISSN 1532-4435. URL http://portal.acm.org/citation. cfm?id=944919.944975.
  23. W. Greene. Econometric Analysis, 5th Edition. Prentice Hall, 2002.
  24. S. M. Gutstein. Transfer Learning Techniques for Deep Neural Nets. PhD thesis, The University of Texas at El Paso, 2010.
  25. I. Guyon, G. Dror, V. Lemaire, D. Silver, G. Taylor, and D. W. Aha. Analysis of the ijcnn 2011 utl challenge. Neural Networks, In press 2011a.
  26. I. Guyon, G. Dror, V. Lemaire, G. Taylor, and D. W. Aha. Unsupervised and transfer learning challenge. In The 2011 International Join Conference on Neural Networks, pages 793-800, July 2011b.
  27. K. Hayashi et al. Self-measuring similarity for multi-task gaussian process. In ICML 2011 Unsupervised and Transfer Learning Workshop. JMLR W&CP, this volume, 2012.
  28. T. Heskes. Empirical bayes for learning to learn. In P. Langley, editor, Proceedings of the International Conference on Machine Learning (ICML'00), pages 367-374, 2000.
  29. G. E. Hinton and T. J. Sejnowski. Unsupervised learning: foundations of neural com- putation. Computational neuroscience. MIT Press, 1999. ISBN 9780262581684. URL http://books.google.ca/books?id=yj04Y0lje4cC.
  30. A. K. Jain, M. N. Murty, and P. J. Flynn. Data clustering: A review, 1999.
  31. T. Jebara. Multi-task feature and kernel selection for svms. In Proceedings of the Interna- tional Conference on Machine Learning (ICML 04), pages 185-192, 2004.
  32. B. Kulis. Icml tutorial on metric learning, 2010. URL http://www.cs.berkeley.edu/ ~kulis/icml2010_tutorial.htm.
  33. N. Lawrence. Probabilistic non-linear principal component analysis with gaussian process latent variable models. JMLR, 6:1783-1816, 2005.
  34. Y.-A. Le Borgne. Supervised dimensionality reduction in the unsupervised and transfer learning 2011 competition, 2011.
  35. G. Lee, L. Stoolman, and C. Scott. Transfer learning for auto-gating of flow cytometry data. In ICML 2011 Unsupervised and Transfer Learning Workshop. JMLR W&CP, this volume, 2012.
  36. H. Lee, A. Battle, R.t Raina, and A. Y. Ng. Efficient sparse coding algorithms. In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Pro- cessing Systems 19, pages 801-808. MIT Press, Cambridge, MA, 2007.
  37. C. Liu, J. Xie, Hui Xiong, and Y. Ge. Stochastic unsupervised learning on unlabeled data. In ICML 2011 Unsupervised and Transfer Learning Workshop. JMLR W&CP, this volume, 2012.
  38. L. Pratt (Editor). Reuse of neural networks through transfer. Connection Science, 8(2), 1996.
  39. U. Luxburg. A tutorial on spectral clustering. Statistics and Computing, 17:395-416, December 2007. ISSN 0960-3174. doi: 10.1007/s11222-007-9033-z. URL http://portal. acm.org/citation.cfm?id=1288822.1288832.
  40. G. Mesnil et al. Unsupervised and transfer learning challenge: a deep learning approach. In ICML 2011 Unsupervised and Transfer Learning Workshop. JMLR W&CP, this volume, 2012.
  41. T. M. Mitchell. The need for biases in learning generalizations. Readings in Machine Learning, pages 184-191, 1980. ed. Jude W. Shavlik and Thomas G. Dietterich.
  42. A. Niculescu-Mizil and R. Caruana. Inductive transfer for bayesian network structure learn- ing. Journal of Machine Learning Research -Proceedings Track, 2:339-346, 2007.
  43. A. Niculescu-Mizil and R. Caruana. Inductive transfer for bayesian network structure learn- ing. In ICML 2011 Unsupervised and Transfer Learning Workshop. JMLR W&CP, this volume, 2012.
  44. V. Nikulin and T.-H. Huang. Unsupervised dimensionality reduction via gradient-based matrix factorization with two adaptive learning rates. In ICML 2011 Unsupervised and Transfer Learning Workshop. JMLR W&CP, this volume, 2012. NIPS1995 Workshop. Learning to learn. http://plato.acadiau.ca/courses/comp/dsilver/ NIPS95 LTL/transfer.workshop.1995.html, 1995.
  45. NIPS2005 Workshop. Inductive transfer -10 years later. http://iitrl.acadiau.ca/itws05/, 2005.
  46. S. J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on Knoweledge and Data Engineering, 22(10):1345-1359, October 2010.
  47. L. Pratt and S. Thrun (Editors). Transfer in inductive systems. Machine Learning, 28(1), 1997.
  48. R. Raina, A. Battle, H. Lee, B. Packer, and A. Y. Ng. Self-taught learning: Transfer learning from unlabeled data. In Proceedings of the Twenty-fourth International Conference on Machine Learning, 2007a.
  49. R. Raina, A. Battle, H. Lee, B. Packer, and A. Y. Ng. Self-taught learning: transfer learning from unlabeled data. In Proceedings of the 24th international conference on Machine learning, ICML '07, pages 759-766, New York, NY, USA, 2007b. ACM. ISBN 978-1-59593-793-3. doi: 10.1145/1273496.1273592. URL http://doi.acm.org/10.1145/ 1273496.1273592.
  50. J. Ramon, K. Driessens, and T. Croonenborghs. Transfer learning in reinforcement learning problems through partial policy recycling. Proc. 18th European Conf. Machine Learning (ECML 2007), pages 699-707, 2007.
  51. M. Saeed. Use of representations in high dimensional spaces for unsupervised and transfer learning challenge, 2011.
  52. R. Salakhutdinov, J. Tenenbaum, and A. Torralba. One-shot learning with a hierarchi- cal nonparametric bayesian model. In ICML 2011 Unsupervised and Transfer Learning Workshop. JMLR W&CP, this volume, 2012.
  53. D. L. Silver and K. P. Bennett. Special issue on inductive transfer. Machine Learning, 73, 2008.
  54. D. L. Silver and R. E. Mercer. The parallel transfer of task knowledge using dynamic learning rates based on a measure of relatedness. Connection Science Special Issue: Transfer in Inductive Systems, 8(2):277-294, 1996.
  55. D. L. Silver, R. Poirier, and D. Currie. Inductive transfer with context-sensitive neural networks. Machine Learning, 73:323-336, 2008.
  56. A. J. Smola, S. Mika, B. Schölkopf, and R. C. Williamson. Regularized principal manifolds. JMLR, 1:179-209, 2001.
  57. M.E. Taylor and P. Stone. Cross-domain transfer for reinforcement learning. Proc. 24th International Conf. Machine Learning (ICML 2007), pages 879-886, 2007.
  58. S. Thrun and L. Y. Pratt (Editors). Learning To Learn. Kluwer Academc Publisher, Boston, MA, 1997.
  59. P. E. Utgoff. Machine Learning of Inductive Bias. Kluwer Academc Publisher, Boston, MA, 1986.
  60. U. von Luxburg, R. C. Williamson, and I. Guyon. Clustering: Science or art? In ICML 2011 Unsupervised and Transfer Learning Workshop. JMLR W&CP, this volume, 2012.
  61. C. Widmer and G. Rätsch. Multitask learning in computational biology. In ICML 2011 Unsupervised and Transfer Learning Workshop. JMLR W&CP, this volume, 2012.
  62. A. Wilson, A. Fern, and P. Tadepalli. Transfer learning in sequential decision problems: A hierarchical bayesian approach. In ICML 2011 Unsupervised and Transfer Learning Workshop. JMLR W&CP, this volume, 2012.
  63. X.-M. WU, A. Man-Cho So, Z. Li, and S.-Y. R. Li. Fast graph laplacian regularized kernel learning via semidefinite quadratic linear programming. In Y. Bengio, D. Schuurmans, J. Lafferty, C. K. I. Williams, and A. Culotta, editors, Advances in Neural Information Processing Systems 22, pages 1964-1972. Curran Associates, Inc., 2009.
  64. Z. Xu et al. Rapid feature learning with stacked linear denoisers, 2011.
  65. A. Zellner. An efficient method for estimating seemingly unrelated regression equations and tests for aggregation bias. Journal of the American Statistical Association, 57:348-368, 1962.

Related papers

ICML2011 Unsupervised and Transfer Learning Workshop
Vincent Lemaire

We organized a data mining challenge in "unsupervised and transfer learning" (the UTL challenge) followed by a workshop of the same name at the ICML 2011 conference in Bellevue, Washington 1. This introduction presents the highlights of the outstanding contributions that were made, which are regrouped in this issue of JMLR W&CP. Novel methodologies emerged to capitalize on large volumes of unlabeled data from tasks related (but different) from a target task, including a method to learn data kernels (similarity measures) and new deep architectures for feature learning.

View PDFchevron_right
Unsupervised and transfer learning challenge
Vincent Lemaire

The 2011 International Joint Conference on Neural Networks, 2011

We organized a data mining challenge in "unsupervised and transfer learning" (the UTL challenge), in collaboration with the DARPA Deep Learning program. The goal of this year's challenge was to learn good data representations that can be re-used across tasks by building models that capture regularities of the input space. The representations provided by the participants were evaluated by the organizers on supervised learning "target tasks", which were unknown to the participants. In a first phase of the challenge, the competitors were given only unlabeled data to learn their data representation. In a second phase of the challenge, the competitors were also provided with a limited amount of labeled data from "source tasks", distinct from the "target tasks". We made available large datasets from various application domains: handwriting recognition, image recognition, video processing, text processing, and ecology. The results indicate that learned data representation yield results significantly better than what can be achieved with raw data or data preprocessed with standard normalizations and functional transforms. The UTL challenge is part of the IJCNN 2011 competition program 1. The website of the challenge remains open for submission of new methods beyond the termination of the challenge as a resource for students and researchers 2 .

View PDFchevron_right
Unsupervised and transfer learning challenge: a deep learning approach
Yoshua Bengio

Unsupervised and Transfer Learning Workshop, in conjunction with ICML, 2011

Learning good representations from a large set of unlabeled data is a particularly challenging task. Recent work (see Bengio (2009) for a review) shows that training deep architectures is a good way to extract such representations, by extracting and disentangling gradually higher-level factors of variation characterizing the input distribution. In this paper, we describe different kinds of layers we trained for learning representations in the setting of the Unsupervised and Transfer Learning Challenge. The strategy of our team won the ...

View PDFchevron_right
Self-taught learning: transfer learning from unlabeled data
Benjamin Packer

2007

We present a new machine learning framework called "self-taught learning" for using unlabeled data in supervised classification tasks. We do not assume that the unlabeled data follows the same class labels or generative distribution as the labeled data. Thus, we would like to use a large number of unlabeled images (or audio samples, or text documents) randomly downloaded from the Internet to improve performance on a given image (or audio, or text) classification task. Such unlabeled data is significantly easier to obtain than in typical semi-supervised or transfer learning settings, making selftaught learning widely applicable to many practical learning problems. We describe an approach to self-taught learning that uses sparse coding to construct higher-level features using the unlabeled data. These features form a succinct input representation and significantly improve classification performance. When using an SVM for classification, we further show how a Fisher kernel can be learned for this representation.

View PDFchevron_right
Analysis of the IJCNN 2011 UTL challenge
Vincent Lemaire

Neural networks : the official journal of the International Neural Network Society, 2012

We organized a challenge in "Unsupervised and Transfer Learning": the UTL challenge (http://clopinet.com/ul). We made available large datasets from various application domains: handwriting recognition, image recognition, video processing, text processing, and ecology. The goal was to learn data representations that capture regularities of an input space for re-use across tasks. The representations were evaluated on supervised learning "target tasks" unknown to the participants. The first phase of the challenge was dedicated to "unsupervised transfer learning" (the competitors were given only unlabeled data). The second phase was dedicated to "cross-task transfer…

View PDFchevron_right
580 California St., Suite 400
San Francisco, CA, 94104
© 2025 Academia. All rights reserved