Quality Estimation

Inferring  evaluation  scores based on human judgments is invaluable compared to using current evaluation metrics which  are not suitable for real-time applications  e.g. post-editing.  However,  these judgments are much more expensive to collect  especially from expert translators, compared to  evaluation based on  indicators contrasting source and  translation texts.  This work  introduces a novel approach for quality estimation by combining learnt confidence scores from a probabilistic inference model based on human judgments, with  selective linguistic features based scores, where the proposed inference model infers the credibility of given human ranks  to solve the  scarcity  and inconsistency issues of human judgments.  Experimental results, using challenging language-pairs, demonstrate improvement in correlation with human judgments over traditional  evaluation metrics.

The vast majority of Machine Translation (MT) evaluation approaches are based on the idea that the closer the MT output is to a human reference translation, the higher its quality. While translation quality has two important aspects, adequacy and fluency, the existing reference-based metrics are largely focused on the former. In this work we combine our metric UPF-Cobalt, originally presented at the WMT15 Metrics Task, with a number of features intended to capture translation fluency. Experiments show that the integration of fluency-oriented features significantly improves the results, rivalling the best-performing evaluation metrics on the WMT15 data.

OpenStreetMap (OSM) is a free spatial data source based on crowd sourced data. Although the OSM data have a range of applications, such as generating 3D models, and routing and navigation, quality issues are still significant concerns when using the data. Several studies have undertaken quality assessments by comparing OSM data with reference data. However, reference data are not always available due to high costs or licensing restrictions, and very few studies have quantitatively estimated the quality of OSM data under conditions where the corresponding reference data are not available. This study proposed the use of a building density (or building coverage ratio) indicator as a proxy, and designed a series of experiments involving different study areas to quantitatively explore the relationship between building density and building completeness for OSM data in urban areas. The residuals (estimated building completeness and reference building completeness) were also analyzed. Two main results were found from the experiments. (1) There was an approximate linear relationship between building density and building completeness in the OSM data. More precisely, the building completeness of OSM data was approximately 3.4-4 times the building density of OSM data. (2) Approximately 70-80% of the absolute residuals were smaller than 10%, and 80-90% of them were smaller than 20%. This shows that, in most cases, estimated building completeness was close to the corresponding reference building completeness. Therefore, we concluded that the building density indicator is a potential proxy for the quantitative completeness estimation of OSM building data in urban areas. The limitations of using this indicator were also addressed.

This submission investigates alternative machine learning models for predicting the HTER score on the sentence level. Instead of directly predicting the HTER score, we suggest a model that jointly predicts the amount of the 4 distinct post-editing operations, which are then used to calculate the HTER score. This also gives the possibility to correct invalid (e.g. negative) predicted values prior to the calculation of the HTER score. Without any feature exploration, a multi-layer perceptron with 4 outputs yields small but significant improvements over the baseline.

Quality Estimation (QE) predicts the quality of machine translation output without the need for a reference translation. This quality can be defined differently based on the task at hand. In an attempt to focus further on the adequacy and informativeness of translations, we integrate features of semantic similarity into QuEst, a framework for QE feature extraction. By using methods previously employed in Semantic Textual Similarity (STS) tasks, we use semantically similar sentences and their quality scores as features to estimate the quality of machine translated sentences. Preliminary experiments show that finding semantically similar sentences for some datasets is difficult and time-consuming. Therefore, we opt to start from the assumption that we already have access to semantically similar sentences. Our results show that this method can improve the prediction of machine translation quality for semantically similar sentences.

Abstract—In this paper we use the statistics provided by a field experiment to explore the utility of supplying machine translation suggestions in a computer-assisted translation (CAT) environment. Regression models are trained for each user in order to estimate the time to edit (TTE) for the current translation segment. We use a combination of features from the current segment and aggregated features from formerly translated segments selected with content-based filtering approaches commonly used in recommendation systems. We present and evaluate decision function heuristics to determine if machine translation output will be useful for the translator in the given segment. We find that our regression models do a reasonable job for some users in predicting TTE given only a small number of training examples; although noise in the actual TTE for seemingly similar segments yields large error margins. We propose to include the estimation of TTE in CAT recommendation systems as a well-corr...