200 Questions About Transfer Learning and Transformers

European Journal of Technology

Purpose: The purpose of this study is to address the limited use of transfer learning techniques in radio frequency machine learning and to propose a customized taxonomy for radio frequency applications. The aim is to enable performance gains, improved generalization, and cost-effective training data solutions in this specific domain. Methodology: The research design employed in this study involves a comprehensive review of existing literature on transfer learning in radio frequency machine learning. The researchers collected relevant papers from reputable sources and analyzed them to identify patterns, trends, and insights. The method of data collection primarily relied on examining and synthesizing existing literature. Data analysis involved identifying key findings and developing a customized taxonomy for radio frequency applications. Findings: The study's findings highlight the limited utilization of transfer learning techniques in radio frequency machine learning. While tra...

Transfer learning (TL), have been becoming extremely popular due to its appalling property of being able to work at a different domain, where it's not trained. Often training at such domain is very costly due to various reasons such as lack of sufficient data etc. Recently Deep Neural Network (DNN) based models have been widely used in the context of TL. These networks as clear transferable layers representing common latent features, hence intuitive to transfer. In this paper we survey various DNN architectures, different training methods adopted to train those and their performance in several application domains.

2019

In this paper, we present a study of the recent advancements which have helped bring Transfer Learning to NLP through the use of semi-supervised training. We discuss cutting-edge methods and architectures such as BERT, GPT, ELMo, ULMFit among others. Classically, tasks in natural language processing have been performed through rule-based and statistical methodologies. However, owing to the vast nature of natural languages these methods do not generalise well and failed to learn the nuances of language. Thus machine learning algorithms such as Naive Bayes and decision trees coupled with traditional models such as Bag-of-Words and N-grams were used to usurp this problem. Eventually, with the advent of advanced recurrent neural network architectures such as the LSTM, we were able to achieve state-of-the-art performance in several natural language processing tasks such as text classification and machine translation. We talk about how Transfer Learning has brought about the well-known Im...

ArXiv, 2020

One desired capability for machines is the ability to transfer their knowledge of one domain to another where data is (usually) scarce. Despite ample adaptation of transfer learning in various deep learning applications, we yet do not understand what enables a successful transfer and which part of the network is responsible for that. In this paper, we provide new tools and analyses to address these fundamental questions. Through a series of analyses on transferring to block-shuffled images, we separate the effect of feature reuse from learning low-level statistics of data and show that some benefit of transfer learning comes from the latter. We present that when training from pre-trained weights, the model stays in the same basin in the loss landscape and different instances of such model are similar in feature space and close in parameter space.

arXiv (Cornell University), 2023

Transformer is a deep neural network that employs a self-attention mechanism to comprehend the contextual relationships within sequential data. Unlike conventional neural networks or updated versions of Recurrent Neural Networks (RNNs) such as Long Short-Term Memory (LSTM), transformer models excel in handling long dependencies between input sequence elements and enable parallel processing. As a result, transformer-based models have attracted substantial interest among researchers in the field of artificial intelligence. This can be attributed to their immense potential and remarkable achievements, not only in Natural Language Processing (NLP) tasks but also in a wide range of domains, including computer vision, audio and speech processing, healthcare, and the Internet of Things (IoT). Although several survey papers have been published highlighting the transformer's contributions in specific fields, architectural differences, or performance evaluations, there is still a significant absence of a comprehensive survey paper encompassing its major applications across various domains. Therefore, we undertook the task of filling this gap by conducting an extensive survey of proposed transformer models from 2017 to 2022. Our survey encompasses the identification of the top five application domains for transformer-based models, namely: NLP, Computer Vision, Multi-Modality, Audio and Speech Processing, and Signal Processing. We analyze the impact of highly influential transformer-based models in these domains and subsequently classify them based on their respective tasks using a proposed taxonomy. Our aim is to shed light on the existing potential and future possibilities of transformers for enthusiastic researchers, thus contributing to the broader understanding of this groundbreaking technology.

2008

Abstract. In order to claim fully general intelligence in an autonomous agent, the ability to learn is one of the most central capabilities. Classical machine learning techniques have had many significant empirical successes, but large real-world problems that are of interest to generally intelligent agents require learning much faster (with much less training experience) than is currently possible.

We organized a data mining challenge in \unsupervised and transfer learning" (the UTL challenge) followed by a workshop of the same name at the ICML 2011 conference in Bellevue, Washington. This introduction presents the highlights of the outstanding con- tributions that were made, which are regrouped in this issue of JMLR W&CP. Novel methodologies emerged to capitalize on large volumes of unlabeled data from tasks related (but di�erent) from a target task, including a method to learn data kernels (similarity measures) and new deep architectures for feature learning.

Lecture Notes in Computer Science, 2008

In this paper, we propose a novel graph-based method for knowledge transfer. We model the transfer relationships between source tasks by embedding the set of learned source models in a graph using transferability as the metric. Transfer to a new problem proceeds by mapping the problem into the graph, then learning a function on this graph that automatically determines the parameters to transfer to the new learning task. This method is analogous to inductive transfer along a manifold that captures the transfer relationships between the tasks. We demonstrate improved transfer performance using this method against existing approaches in several real-world domains.

We organized a data mining challenge in "unsupervised and transfer learning" (the UTL challenge) followed by a workshop of the same name at the ICML 2011 conference in Bellevue, Washington 1. This introduction presents the highlights of the outstanding contributions that were made, which are regrouped in this issue of JMLR W&CP. Novel methodologies emerged to capitalize on large volumes of unlabeled data from tasks related (but different) from a target task, including a method to learn data kernels (similarity measures) and new deep architectures for feature learning.

IEEE Transactions on Emerging Topics in Computational Intelligence, 2019