From July 13 to August 15, 2014, the CRITT plans to conduct a Translation Data Analytics (TDA) project.
The aim of the TDA project is to explore and analyse user activity data which is collected in advanced man-machine communication situations. The TDA project will assess and elaborate methods to produce data-driven user profiles, to investigate differences in communication styles, and to identify patterns of user behavior for more and less successful man-machine communication. Data repositories such as the CRITT TPR-DB will be taken as a basis to analyze concrete and specialized forms of professional man-machine communication such as translators behaviour in advanced computer-assisted collaborative production environments.
An introductory PhD summer course on Translation Process Research (TPR) and a one-day workshop will precede the TDA project, in which students get acquainted with peculiarities of the TPR data and data acquisition methods. The TDA project will use exploratory statistical approaches for discovering new features and dependencies in the TPR data, and to formulate hypotheses about the causal relations. Statistical hypothesis tests will be deployed for confirming or falsifying existing hypotheses.
After four weeks of intense work on the TDA project, we will present preliminary results during four morning sessions from August 11 to August 14 in the Mødehuset, Spejdercenter Holmen (http://www.spejdercenterholmen.dk/velkommen).
Annegret Sturm, Bergljot Behrens, Moritz Schaeffer, Arndt Heilmann, Maheshwar Ghankot:
Abstract: The present study investigates the question whether the co-activation of both source and target language have an influence on the translator's behaviour. A way to measure co-activation is the comparative analysis of the influence of different syntactic realizations of the target language entropy on gaze time and production duration during translation and post-editing. We measure syntactic choice in terms of entropy which quantifies the distribution of different translations realisations of a given source segment. High entropy is an indicator of selection effort related to available options of the final translation out of many different realizations. We investigate the impact of syntactic entropy on cognitive effort. In a first step, as research by Jensen et al. (2010) su ggests, source text segments which need reordering yield longer gaze times than segments which do not need reordering. In a second step, based on the assumption that syntax is shared across languages (Hartsuiker & Pickering 2004), a recently activated syntactic structure is likely to influence subsequent processing, thus “priming” it. Low syntactic entropy of translation choices could, thus, be taken to be a sign of priming. To test the hypothesis whether syntactic choices have an influence on cognitive effort, we compared four datasets comprising translation and post-editing data of the same English source texts translated into Danish, German, Spanish and Hindi. Data was manually annotated for syntactic structure along three relevant features: clause type, valency and voice. Our analyses reveal a positive correlation between syntactic entropy of translation realizations values and gaze as well as production time in all four languages. However, no effect of syntactic entropy could be detected in the post-editing data, suggesting that the post-editors were primed by the MT output.
Abstract: I tried to predict gaze fixation duration at source text using supervised learning. GazeS basically represents the sum total of all fixation on a particular source text word during the whole task, it summarize the visual attention a particular source text word has received during the whole task. It can be described as Machine Learning models for predicting the gaze fixation duration on source text using lexical, syntactic and semantic information. For this experiment I have used TPR-DB (KTHJ08) dataset which has been prepared and maintained by CRITT, CBS. Different combinations of feature has been extracted from the dataset and different model built. In my experiment the dependant variable was Gaze at source, and the independent variable was different feature. Then train the model into support vector machine. After training that model different classification accuracy achieved. So far I get close to 50% of classification accuracy with the baseline of 25% classification accuracy. From the several sets of experiment my findings is that unigram, bigram frequency of the source word, length of the word, Translator identity, perplexity, lexical entropy, syntantic entropy, polysemy of word, SuperTag are the very good predictor of Gaze at source.
Karan Singla, David Orrego Carmona, Ashleigh Gonzales:
Abstract: Computer-assisted translation remains a progressive field of research, and there is an evergrowing interest in providing translators and post-editors with better software tools to facilitate their work and increase productivity. The purpose of the current investigation is to predict post-editor profiles based on user behaviour and demographics using novel machine learning techniques to gain a better understanding of post-editor styles. Our study extracts process unit features from the CasMaCat LS14 database from the CRITT Translation Process Research Database (TPR-DB). The analysis has two main research goals: We create n-gram models based on user activity and part-of-speech sequences to automatically cluster post-editors, and we use discriminative classifier models to characterize post-editors based on a diverse range of translation process features. Our results allow for more targeted consideration of requirements unique to post-editors in user interface design of a translator’s workbench.
Akshay Minocha, Alena Konina:
Abstract: The analytics of the translation process research depends on many factors, and in our case, with the TPR-DB, there is a lot of untapped potential in this research field that we would like to look into, and some of it we did in this research. Our aim is to identify the areas of interest for translators while translating a source text into target text. This research shows and promises to predict the parts of a sentence where the attention of the translator is for a specific time while translating a chunk, the implications of this would let us know, which parts of the sentence are more difficult to understand, how the patterns change across different translators and different studies involving many target text languages, predicting the look ahead buffer which would help translation systems and other machine translation and speech recognition tools by focussing their interest on this particular area.
Tatiana Fedorchenko, Arlene Koglin, Bartolomé Mesa-Lao, Mercedes García Martinez, Julián Zapata:
Abstract: Our group looked into the user activity data collected in two different studies, i.e. LS14 and CFT14, in the CRITT Translation Process Research Database. Both datasets were collected in the framework of the CasMaCat project. LS14 is a longitudinal study involving 5 post-editors working with interactive machine translation over a period of six week. The aim was to know whether they are faster working with interactivity as they become acquainted with this type of assistive technology. Results show that participants became faster over time after working with interactivity during the post-editing process (week 1 - week 5). Baseline: traditional post-editing vs. interactive machine translation. However, there was an unexpected increase in time in week 6 that could be explained due to the quality of the MT provided during this last week. CFT14 dataset originates in the third CasMaCat field trial. The aim of this study was to explore the benefits of working with interactive machine translation combined with online learning techniques for post-editing purposes. Baseline: traditional post-editing vs. interactive machine translatoin with online learning. As it should be expected, results show that working with online learning techniques made the post-editing process faster, but only when the time used by the post-editors to make Internet searches is not taken into account. Our analyses make clear that productivity metrics in terms of overall time to complete the task might not be a good indicator of performance when the post-editor needs to conduct Internet searches in order to verify the quality of the MT provided.
Andreas Søeborg Kirkedal, Dipti Pandey, Julián Zapata:
This work is divided into two parts. First, we present a
pilot study that investigates the usefulness of automatic speech
recognition (ASR) as an input method for translation and post-editing in
a native or a foreign language. The pilot study uses different hardware
and software configurations as well as Spanish native and English and
French non-native speakers in the experiments. The experiments were a
monolingual dictation setting (i.e., the mental translation process was
not involved). The pilot study concludes that not only native speakers
but also non-native speakers with a lower recognition accuracy could
benefit from the use of speech modality in translation and
post-editing. Even after 2 iterations of dictation with commands and
manual post-editing of ASR output, the participants still showed
improvements w.r.t. to time duration. This leads to a tentative lower
bound on the recognition accuracy for useful ASR in dictation settings.
Pintu Lohar, Ambarish Jadhav:
Abstract: My main goal was to clean up my data and to get it into an analyzable format. I worked on data which I gathered as part of my PhD project: 72 translation and post-editing sessions from master's students of translation logged with Casmacat, Inputlog and an EyeLink eyetracker. The main challenges were merging the EyeLink and Casmacat data (which was successful), and mering the Inputlog and Casmacat data (which turned out to be harder than anticipated). Other than preparing the data, I also looked into the usage of external resources during translation and post-editing. I found that concordancers and dictionaries were the most frequently used tools, along with Google search (which was often a way to get to other sources rather than a source in itself). Both concordancers and dictionaries were used more frequently when translating than when post-editing, though closer inspection revealed that this also depends on the participant as well as the text. More research is needed to identify exactly why these participants or these texts differ with regards to external resources consulted. Extra factors that will be taken into account are the time spent looking at external resources, meta-data relating to participants' attitude towards machine translation and post-editing, the type of search queries, text topics, structure and vocabulary of the source text, and quality of the machine translation. In a later stage, information on external resources will be linked to the final translation quality and productivity of the participants.
The idea of the TDA project is to approach the CRITT TPR-DB from different possible angles. In order to do so, we have formulated a few challenges. Participants are invited to add further challenges according to their interests, or to extend/modify the list below:
In order to investigate these and similar questions, we will examine the TPR-DB which consists of richly annotated reading, writing, translation and post-editing data. Besides gaze-, keystroke-, and word alignment data, some parts of the data also contain quality information of reviewer and evaluators, as well as meta information about the translator/post-editor experience.
Information from the TPR-DB will be used for
this task and a set of gold standard gaze-to-word mappings will be
provided for training and evaluation. Success of different approaches
will be evaluated by comparing the predicted gaze paths and word
mappings to the gold standard using the Levenshtein distance, or
Precision and Recall scores.
the task is to predict the next keystroke (at time t+1)
that the translator or post-editor is going to produce. Success of
participants will be measured by comparing the identity and the time of
the predicted keystrokes with respect to the actual keystrokes produced
by human translators.
Location: The TDA project will take place in the Spejdercenter Holmen, Arsenalvej 10, 1436, København, Denmark.
Accommodation: dormitory accommodation for up to 10 students can be provided in the Spejdercenter Holmen.
Structure: According to the interests of each participant, we will define small tasks and sub-groups to work on a focussed sub-project and discuss its relation to the TDA project on a daily basis. Similar to last year SEECAT project, we'll have lectures and discussion sessions in the morning and team work in the afternoons.
Application: Participation is free of charge. A number of grants for accommodation and travel are available for students and early career researchers. Applications (cover letter + full CV) must be sent to Michel Carl (firstname.lastname@example.org) before March 15, 2014.