Temporal Classification

We present work on linking events and fluents (i.e., relations that hold for certain periods of time) to temporal information in text, which is an important enabler for many applications such as timelines and reasoning. Previous research has mainly focused on temporal links for events, and we extend that work to include fluents as well, presenting a common methodology for linking both events and relations to timestamps within the same sentence. Our approach combines tree kernels with classical feature-based learning to exploit context and achieves competitive F1-scores on event-time linking, and comparable F1scores for fluents. Our best systems achieve F1-scores of 0.76 on events and 0.72 on fluents. * The first author conducted this research during an internship at IBM Research.

Background: Language is one of the first faculties afflicted by Alzheimer's disease (AD). A growing body of work has focussed on leveraging automated analysis of speech to accurately predict the onset of AD. Previous work, however, did not address the effects of AD on the structure of discourse in spontaneous speech and literature. Aims: Our goal is to identify the effects of AD on the structure of discourse, both in spontaneous speech and in literature. Methods & Procedures: We use two existing data sets, DementiaBank and the Carolina Conversations Collection, to explore how AD manifests itself in spontaneous speech. This is done by automatically extracting discourse relations according to Rhetorical Structure Theory. We also study written novels, comparing authors with and without dementia using the same tools. Outcomes & Results: Several discourse relations, especially those involving elaboration and attribution, are significant indicators of AD in speech. Indicators of the disease in written text, by contrast, involve relations of logical contingency. Conclusions: Our work highlights how AD can alter discourse structures in both spontaneous speech and written text. Future work should combine discourse analysis with previously studied lexico-syntactic features.

In this paper we classify the temporal relations between pairs of events on an article-wide basis. This is in contrast to much of the existing literature which focuses on just event pairs which are found within the same or adjacent sentences. To achieve this, we leverage on discourse analysis as we believe that it provides more useful semantic information than typical lexico-syntactic features. We propose the use of several discourse analysis frameworks, including 1) Rhetorical Structure Theory (RST), 2) PDTB-styled discourse relations, and 3) topical text segmentation. We explain how features derived from these frameworks can be effectively used with support vector machines (SVM) paired with convolution kernels. Experiments show that our proposal is effective in improving on the state-of-the-art significantly by as much as 16% in terms of F 1 , even if we only adopt less-than-perfect automatic discourse analyzers and parsers. Making use of more accurate discourse analysis can further boost gains to 35%.

Slot-filling models in task-driven dialog systems rely on carefully annotated training data. However, annotations by crowd workers are often inconsistent or contain errors. Simple solutions like manually checking annotations or having multiple workers label each sample are expensive and waste effort on samples that are correct. If we can identify inconsistencies, we can focus effort where it is needed. Toward this end, we define six inconsistency types in slot-filling annotations. Using three new noisy crowd-annotated datasets, we show that a wide range of inconsistencies occur and can impact system performance if not addressed. We then introduce automatic methods of identifying inconsistencies. Experiments on our new datasets show that these methods effectively reveal inconsistencies in data, though there is further scope for improvement.

Slot-filling models in task-driven dialog systems rely on carefully annotated training data. However, annotations by crowd workers are often inconsistent or contain errors. Simple solutions like manually checking annotations or having multiple workers label each sample are expensive and waste effort on samples that are correct. If we can identify inconsistencies, we can focus effort where it is needed. Toward this end, we define six inconsistency types in slot-filling annotations. Using three new noisy crowd-annotated datasets, we show that a wide range of inconsistencies occur and can impact system performance if not addressed. We then introduce automatic methods of identifying inconsistencies. Experiments on our new datasets show that these methods effectively reveal inconsistencies in data, though there is further scope for improvement.

We study the problem of classifying the temporal relationship between events and time expressions in text. In con-trast to previous methods that require extensive feature engineering, our ap-proach is simple, relying only on a measure of parse tree similarity. Our method generates such tree similarity values using dependency parses as in-put to a convolution kernel. The re-sulting system outperforms the current state-of-the-art. To further improve classifier performance, we can obtain more annotated data. Rather than rely on expert annotation, we assess the feasibility of acquiring annotations through crowdsourcing. We show that quality temporal relationship annota-tion can be crowdsourced from novices. By leveraging the problem structure of temporal relation classification, we can selectively acquire annotations on problem instances that we assess as more difficult. Employing this anno-tation strategy allows us to achieve a classification accuracy of 73.2%, a statistically signific...

This paper introduces the first pattern-based Persian Temporal Relation Classifier (PTRC) that finds the type of temporal relations between pairs of events in the Persian texts. The proposed system uses support vector machines (SVMs) equipped by combinations of simple, convolution tree, and string subsequence kernels (SSK). In order to evaluate the algorithm, we have developed a Persian TimeBank (PTB) corpus. PTRC not only increases the performance of the classification by applying new features and SSK, but also alleviates the probable adverse effects of the Free Word Orderness (FWO) of Persian on temporal relation classification. We have also applied our proposed algorithm to two standard corpora on English (i.e., TimeBank and TempEval-2) to measure the efficiency of the new features and SSK. The experiments show the accuracies of 65.6%, 59.53%, 50.2%, and 62.17% on an augmented version of PTB, TimeBank, tasks E and F of TempEval-2, respectively. Consequently, we have achieved the ...

Discovery of temporal information is key for organising knowledge and therefore the task of extracting and representing temporal information from texts has received an increasing interest. In this paper we focus on the discovery of temporal footprints from encyclopaedic descriptions. Temporal footprints are time-line periods that are associated to the existence of specific concepts. Our approach relies on the extraction of date mentions and prediction of lower and upper boundaries that define temporal footprints. We report on several experiments on persons' pages from Wikipedia in order to illustrate the feasibility of the proposed methods.

We present a sequential model for temporal relation classification between intrasentence events. The key observation is that the overall syntactic structure and compositional meanings of the multi-word context between events are important for distinguishing among fine-grained temporal relations. Specifically, our approach first extracts a sequence of context words that indicates the temporal relation between two events, which well align with the dependency path between two event mentions. The context word sequence, together with a parts-of-speech tag sequence and a dependency relation sequence that are generated corresponding to the word sequence, are then provided as input to bidirectional recurrent neural network (LSTM) models. The neural nets learn compositional syntactic and semantic representations of contexts surrounding the two events and predict the temporal relation between them. Evaluation of the proposed approach on TimeBank corpus shows that sequential modeling is capable of accurately recognizing temporal relations between events, which outperforms a neural net model using various discrete features as input that imitates previous feature based models. Consider the example sentence in Figure 1. Here, the first two temporal re

Question Answering (QA) system is a field of Natural language processing, which allows users to ask questions using the natural language sentence and return a brief answer to the users rather than a list of documents. One type of QA systems is Temporal Question Answering system. Time is an important dimension in any information system. Time-specific information embedded in document in the form of temporal expressions. Temporal expressions are words or phrases that describe a point, duration in time. Extraction of these expressions from raw text is used for a wide range of information retrieval tasks such as Summarization, Question Answering and other tasks. In this paper we give a survey of Temporal Question Answering system which focuses on answering temporal questions posted by humans in natural language.

Background: Language is one of the first faculties afflicted by Alzheimer's disease (AD). A growing body of work has focussed on leveraging automated analysis of speech to accurately predict the onset of AD. Previous work, however, did not address the effects of AD on the structure of discourse in spontaneous speech and literature. Aims: Our goal is to identify the effects of AD on the structure of discourse, both in spontaneous speech and in literature. Methods & Procedures: We use two existing data sets, DementiaBank and the Carolina Conversations Collection, to explore how AD manifests itself in spontaneous speech. This is done by automatically extracting discourse relations according to Rhetorical Structure Theory. We also study written novels, comparing authors with and without dementia using the same tools. Outcomes & Results: Several discourse relations, especially those involving elaboration and attribution, are significant indicators of AD in speech. Indicators of the disease in written text, by contrast, involve relations of logical contingency. Conclusions: Our work highlights how AD can alter discourse structures in both spontaneous speech and written text. Future work should combine discourse analysis with previously studied lexico-syntactic features.

Crowdsourcing is an accessible and cost-effective alternative to traditional methods of collecting and annotating data. The application of crowdsourcing to simple tasks has been well investigated. However, complex tasks like semantic annotation transfer require workers to take simultaneous decisions on chunk segmentation and labeling while acquiring on-the-go domain-specific knowledge. The increased task complexity may generate low judgment agreement and/or poor performance. The goal of this paper is to cope with these crowdsourcing requirements with semantic priming and unsupervised quality control mechanisms. We aim at an automatic quality control that takes into account different levels of workers' expertise and annotation task performance. We investigate the judgment selection and aggre-gation techniques on the task of cross-language semantic annotation transfer. We propose stochastic modeling techniques to estimate the task performance of a worker on a particular judgment with respect to the whole worker group. These estimates are used for the selection of the best judgments as well as weighted consensus-based annotation aggregation. We demonstrate that the technique is useful for increasing the quality of collected annotations.

This paper describes two sets of crowdsourcing experiments on temporal information annotation conducted on two languages, i.e., English and Italian. The first experiment, launched on the CrowdFlower platform, was aimed at classifying temporal relations given target entities. The second one, relying on the CrowdTruth metric, consisted in two subtasks: one devoted to the recognition of events and temporal expressions and one to the detection and classification of temporal relations. The outcomes of the experiments suggest a valuable use of crowdsourcing annotations also for a complex task like Temporal Processing.

One of the major bottlenecks in the development of data-driven AI Systems is the cost of reliable human annotations. The recent advent of several crowdsourcing platforms such as Amazon's Mechanical Turk, allowing requesters the access to affordable and rapid results of a global workforce, greatly facilitates the creation of massive training data. Most of the available studies on the effectiveness of crowdsourcing report on English data. We use Mechanical Turk annotations to train an Opinion Mining System to classify Spanish consumer comments. We design three different Human Intelligence Task (HIT) strategies and report high inter-annotator agreement between non-experts and expert annotators. We evaluate the advantages/drawbacks of each HIT design and show that, in our case, the use of non-expert annotations is a viable and cost-effective alternative to expert annotations.

Crowdsourcing platforms are a popular choice for researchers to gather text annotations quickly at scale. We investigate whether crowdsourced annotations are useful when the labeling task requires medical domain knowledge. Comparing a sentence classification model trained with
expert-annotated sentences to the same model trained on crowd-labeled sentences, we find the crowdsourced training data to be just as effective as the manually produced dataset. We can improve the accuracy of the crowd-fueled model with-
out collecting further labels by filtering out worker labels applied with low confidence.

One of the major bottlenecks in the development of data-driven AI Systems is the cost of reliable human annotations. The recent advent of several crowdsourcing platforms such as Amazon's Mechanical Turk, allowing requesters the access to affordable and rapid results of a global workforce, greatly facilitates the creation of massive training data. Most of the available studies on the effectiveness of crowdsourcing report on English data. We use Mechanical Turk annotations to train an Opinion Mining System to classify Spanish consumer comments. We design three different Human Intelligence Task (HIT) strategies and report high inter-annotator agreement between non-experts and expert annotators. We evaluate the advantages/drawbacks of each HIT design and show that, in our case, the use of non-expert annotations is a viable and cost-effective alternative to expert annotations.

This paper reports a crowdsourcing experiment on the identification and classification of event types in Italian. The data collected show that the task is not trivial (360 trusted judgments collected vs. 475 untrsuted ones) but it has been shown to be linguistically felicitous. The overall accuracy of the annotation is 61.6%. A reliability threshold assigned to the workers allows us to indentify the sub-population who has the awareness to perform this complex task and the accuracy of this sub-population is raised to 93%. Our hypothesis is that although the initial crowdsourced data is necessarily noisy, it can yield high quality results if the sub-population of 'good' workers can be identified. In other words, crowdsourcing offers a solution to difficult annotation tasks as long as there is an effective way to identify the reliable workers. TITLE AND ABSTRACT IN ANOTHER LANGUAGE, L 2 (OPTIONAL, AND ON SAME PAGE)

In this paper we classify the temporal relations between pairs of events on an article-wide basis. This is in contrast to much of the exist- ing literature which focuses on just event pairs which are found within the same or adjacent sentences. To achieve this, we leverage on discourse analysis as we believe that it provides more useful semantic information than typical lexico-syntactic features. We propose the use of several discourse analysis frameworks, including 1) Rhetorical Structure Theory (RST), 2) PDTB-styled discourse relations, and 3) topical text segmentation. We explain how features derived from these frameworks can be effectively used with support vector machines (SVM) paired with convolution kernels. Ex- periments show that our proposal is effective in improving on the state-of-the-art significantly by as much as 16% in terms of F, even if we only adopt less-than-perfect automatic discourse analyzers and parsers. Making use of more accurate discourse analysis can further boost gains to 35%.

We study the problem of classifying the temporal relationship between events and time expres- sions in text. In contrast to previous methods that require extensive feature engineering, our approach is simple, relying only on a measure of parse tree similarity. Our method generates such tree similarity values using dependency parses as input to a convolution kernel. The resulting system outperforms the current state-of-the-art.
To further improve classifier performance, we can obtain more annotated data. Rather than rely on expert annotation, we assess the feasibility of acquiring annotations through crowdsourcing. We show that quality temporal relationship annotation can be crowdsourced from novices. By leveraging the problem structure of temporal relation classification, we can selectively acquire annotations on problem instances that we assess as more difficult. Employing this annotation strategy allows us to achieve a classification accuracy of 73.2%, a statistically significant improvement of 8.6% over the previous state-of-the-art, while trimming annotation efforts by up to 37%.
Finally, as we believe that access to sufficient training data is a significant barrier to current temporal relationship classification, we plan to share our collected data with the research community to promote benchmarking and comparative studies.

Abstract In this paper, we develop an RST-style textlevel discourse parser, based on the HILDA discourse parser (Hernault et al., 2010b). We significantly improve its tree-building step by incorporating our own rich linguistic features. We also analyze the difficulty of extending traditional sentence-level discourse parsing to text-level parsing by comparing discourseparsing performance under different discourse conditions.

Abstract Temporal analysis of events is a central problem in computational models of discourse. However, correctly recognizing temporal aspects of events poses serious challenges. This paper introduces a joint modeling framework and feature set for temporal analysis of events that utilizes Markov Logic. The feature set includes novel features derived from lexical ontologies. An evaluation suggests that introducing lexical relation features improves the overall accuracy of temporal relation models.

Recent years have seen increasing attention in temporal processing of texts as well as a lot of standardization effort of temporal information in natural language. A central part of this information lies in the temporal relations between events described in a text, when their precise times or dates are not known. Reliable human annotation of such information is difficult, and automatic comparisons must follow procedures beyond mere precision-recall of local pieces of information, since a coherent picture can only be considered at a global level. We address the problem of evaluation metrics of such information, aiming at fair comparisons between systems, by proposing some measures taking into account the globality of a text.