Active learning and the total cost of annotation

2020

This thesis presents the novel document annotation tool ActiveAnno. The focus of ActiveAnno is to be a well-functioning general-purpose annotation tool for document-level annotations. Through high configurability, users can optimize for high-quality annotations and e ciency. One of the main features to enable this is a machine learning integration with the ability to provide pre-annotations and to integrate ActiveAnno into an active learning process. In this context, two experiments with ActiveAnno were conducted in an industry setting. In the first experiment, the e↵ects of document-level pre-annotations on annotation quality and e ciency were analyzed in comparison to the manual annotation process. The results show a 28% decrease in annotation duration using pre-annotations. At the same time, annotator accuracy and inter-annotator agreement show no significant changes. The second experiment compared model performance for incremental learning with random sampling and active learning with uncertainty sampling for a new annotation task in ActiveAnno, using its machine learning integration capabilities and a fastText-based external classification service. The results show a worse performance for a model trained with active learning when not providing any pre-trained word vectors to it. Using pre-trained word vectors, both, models trained with random sampling and uncertainty sampling perform better overall, without a significant di↵erence between each other. These results show that the choice of classifier is relevant in an active learning setting and that using unsupervised methods in form of pre-trained word vectors can prevent the sampling bias of uncertainty sampling in this context. I Zusammenfassung Zusammenfassung Diese Masterarbeit stellt ein neues Annotationstool für Dokumente namens Ac-tiveAnno vor. Der Fokus von ActiveAnno liegt darauf, ein gut funktionierendes, vielseitig einsetzbares Annotationstool für Annotationen auf Dokumentenebene zu sein. Durch hohe Konfigurierbarkeit können die Nutzer sowohl für hochqualitative Annotationen als auch für einen e zienten Annotationsprozess optimieren. Eine der wichtigsten Funktionen, um dies zu ermöglichen, ist eine Machine Learning Integration mit der Möglichkeit, Pre-Annotationen bereitzustellen sowie ActiveAnno in einen Active Learning Prozess zu integrieren. In diesem Kontext wurden zwei Experimente mit ActiveAnno innerhalb eines Industrieumfelds durchgeführt. Im ersten Experiment wurde der E↵ekt von Pre-Annotationen auf Dokumentenebene mit Bezug auf Qualität der Annotationen und E zienz analysiert und mit dem manuellen Annotationsvorgang verglichen. Die Ergebnisse zeigen eine 28% kürzere Annotationsdauer mit Pre-Annotationen. Gleichzeitig gab es keine signifikanten Anderungen bei der Genauigkeit der Annotatoren oder dem Inter-Annotator Agreement. Das zweite Experiment vergleicht die Performance von Models für Incremental Learning mit Random Sampling und Active Learning mit Uncertainty Sampling für eine neue Annotationsaufgabe in ActiveAnno, unter Nutzung der Möglichkeiten zur Machine Learning Integration und einem auf fastText basierenden, externen Classification Service. Die Ergebnisse zeigen eine schlechtere Performance für ein mit Active Learning trainiertes Model, wenn keine vortrainierten Wort-Vektoren bereitgestellt wurden. Unter Verwendung von vortrainierten Wort-Vektoren performen sowohl Models trainiert mit Random Sampling als auch Uncertainty Sampling besser, ohne, dass es zwischen beiden signifikante Unterschiede gibt. Diese Ergebnisse zeigen, dass die Wahl des Classifiers im Kontext von Active Learning relevant ist und Unsupervised Learning Methoden in der Form von vortrainierten Wort-Vektoren den Sampling Bias von Uncertainty Sampling in diesem Fall verhindern können. II Acknowledgement Acknowledgement This thesis was done in cooperation with the Qualitize GmbH. I want to thank them for their support, for the opportunity to create this open-source tool, and for the ability to use this tool to conduct research on annotation tools in an industry setting. III 1.1. Context samples required. Finally, the newly trained model can be used to automatically annotate either all or a subset of the documents, greatly reducing the manual labor required to finish the annotation task. 1.1 Context During the Web Interfaces for Language Processing Systems project at the University of Hamburg, the author created the document annotation tool ActiveAnno 1 , which was done in cooperation with a local company. ActiveAnno is an open-source annotation tool with focus on document-level annotations. For this master thesis, ActiveAnno was improved and extended to have a machine learning integration and active learning capabilities to pursue the goal of creating high-quality annotations while minimizing e↵ort. The thesis was also done in cooperation with the local company, o↵ering the ability to conduct experiments with annotators, tasks and data from an industry environment. 1.2 Research Questions By leveraging the ability to conduct experiments in a practical setting, two research questions are investigated. Both revolve around using machine learning to improve annotation processes with regards to annotation quality and e ciency. Research Question 1 What is the e↵ect of pre-annotations in the form of generated document-level annotations from a machine learning model on the annotation quality and the e ciency with which these annotations are created? Research Question 2 How does active learning with uncertainty sampling compare to incremental learning with random sampling with regards to the generated annotation quality given a new machine learning model without preexisting training data?

Proceedings of the Linguistic Annotation Workshop on - LAW '07, 2007

In the construction of a part-of-speech annotated corpus, we are constrained by a fixed budget. A fully annotated corpus is required, but we can afford to label only a subset. We train a Maximum Entropy Markov Model tagger from a labeled subset and automatically tag the remainder. This paper addresses the question of where to focus our manual tagging efforts in order to deliver an annotation of highest quality. In this context, we find that active learning is always helpful. We focus on Query by Uncertainty (QBU) and Query by Committee (QBC) and report on experiments with several baselines and new variations of QBC and QBU, inspired by weaknesses particular to their use in this application. Experiments on English prose and poetry test these approaches and evaluate their robustness. The results allow us to make recommendations for both types of text and raise questions that will lead to further inquiry.

Proceedings of the 2007 Joint …, 2007

We consider the impact Active Learning (AL) has on effective and efficient text cor-pus annotation, and report on reduction rates for annotation efforts ranging up until 72%. We also address the issue whether a corpus annotated by means of AL – using a particu-lar ...

Proceedings of the NIPS Workshop on …, 2008

The goal of active learning is to minimize the cost of training an accurate model by allowing the learner to choose which instances are labeled for training. However, most research in active learning to date has assumed that the cost of acquiring labels is the same for all ...