Work Plan‎ > ‎

Spoken Dialogue System Enhancement and Customization


Provide tools and algorithms for the enhancement and customization of deployed speech services, aka service doctoring. Specifically: 1) prompt/grammar enhancement, 2) dialogue flow enhancement, 3) service customization (user modeling), 4) multilingual application service doctoring. The input from IVR analytics WP2 and KPIs will be used to update the speech service components using machine learning algorithms and (fully or partially) transcribed service data/logs.

Description of Work

  • Task 3.1 Prompt and Grammar Enhancement [TSI-TUC, VoiceWeb, NuEcho] Based on KPI performance and input from the IVR analytics module we will select the most appropriate prompts from the pool of prompts available in the application. Using transcribed utterances we will automatically train statistical grammars and update the grammars to improve performance (see also [Suendermann et al 2010]). Finite state machine grammars will be update using the NuGram platform by Nu Echo, enhanced by grammar induction technology by TSI-TUC.
  • Task 3.2 Dialogue Flow Enhancement [KTH, NuEcho] This task will explore different ways of automatically improving dialogue strategies in deployed systems. Using the machine learning algorithms developed in Task 2.2 to identify successful and problematic interactions, together with the annotated data, this task will develop methods of automatic adjustment of the policies of the system that increase the probability of successful interactions. This could be done either with supervised learning or reinforcement learning (MDP or POMDP). The efficiency of these enhancements will be tested in an interactional setting.
  • Task 3.3 User Modeling [KTH] In this task, we will explore the adaptation of the models developed in Task 2.2 and Task 3.3 to specific user populations. This will be achieved via unsupervised learning to cluster users into user types and develop a user type classifier. This information will then be used to improve the hot-spot detection and dialogue flow enhancement algorithms, as well as, customize speech applications to specific user population (e.g., naive vs. experienced users).
  • Task 3.4 Multilinguality [INESC ID, TSI-TUC] We will use statistical machine translation and crowd-sourcing to improve on prompts and grammars for multilingual applications. Corpus-based methods for statistical grammar training and direct translation of service grammars will be used to reduce the enhancement and customization cycle for new languages and multilingual applications.


  • D3.1 Interim Report on SDS Enhancement (M12)
  • D3.2 SDS Enhancement Demonstrator (M16)
  • D3.3 Final Report on SDS Enhancement (M24)