My Work


@ Interaction Lab, HWU

EMOTE is an European Commission funded FP7 project whose objective is to build an empathic robotic tutor to help teachers in schools. Will such a robot engage the students and help improve learning? This is the question we try to answer in this project. My responsibility was to build the interaction manager for the robot. We use a Nao robot, a huge 55 inch touch table and a host of sensors to read the affective state of the users. The touch table is used to display an activity that the student is supposed to work on. We have two activities: a map application and a serious game called Enercities. The robot stand opposite to the student(s) and interacts with them as they work on the activity. In each step of the activity, the student is supposed to exercise his/her skills to solve a problem. Students can get stuck due to their inability to solve the problem. In such cases, the robot senses a timeout or an incorrect answer and tries to help the student with pedagogical tactics. The interaction manager is the key decision maker in the system that decides what to tell the student in a given context.

We have implemented an IM engine that can read dialogue scripts in the form of an XML file. This script will define the dialogue task and strategy of the IM. This division of engine and scripts is very useful because dialogue designers only have to change the script files to modify the dialogue behaviour of the robot or to define a new task. 

SpeechCity (

@ Interaction Lab, HWU

SpeechCity is a commercialisation project funded by EPSRC, UK. In this project, we explored the idea of a multi-tasking conversational assistant on a smartphone for tourists. The focus was more on implementation than on the research questions. We built a conversational tour guide for Edinburgh using open data such as Wikipedia, OpenStreetMaps, OpenWeatherMaps, FourSquare, etc. Users can ask for restaurants, cafes, pubs and other amenities as they are walking along on the streets of Edinburgh. They can also ask for directions to destinations, weather info, and stories about people and places. The novel idea is that it is a multi-tasking conversational agent that can run several conversation threads at the same time and switch between them dynamically as the context changes. It can navigate you to the castle. But when you walk near a point of interest, it will tell you a story about it. And then switch back to the navigation task. You can also ask for more if you are interested in a story or a person. 

We won the EdinburghApps challenge for the culture track organised by the City of Edinburgh Council for the most innovative apps for the city. We contested against several startups and software companies to win this award.


@ Interaction Lab, HWU

SPACEBOOK is a spoken dialogue system that interacts with pedestrian users providing them directions, finding cafes, restaurants, etc and giving users interesting information on landmarks of the city. 

Related publications:

  • Srinivasan Janarthanam, Oliver Lemon, Xingkun Liu, Phil Bartie, William Mackaness, Tiphaine Dalmas and Jana Goetze , 2012, "Integrating location, visibility, and Question-Answering in a spoken dialogue system for pedestrian city exploration", Accepted at SEMDIAL 2012, Paris.
  • Srinivasan Janarthanam, Oliver Lemon, Xingkun Liu, Phil Bartie, William Mackaness, Tiphaine Dalmas and Jana Goetze , 2012, "Conversational Natural Language interaction for Place-related
    Knowledge Acquisition", Accepted at Place-related Knowledge Acquisition Research Workshop (PKAR), Spatial Cognition Conference, Germany.
  • Srinivasan Janarthanam and Oliver Lemon, 2012,  "A Web-based Evaluation Framework for Spatial Instruction-Giving Systems", Accepted at ACL conference 2012, South Korea.
  • Srinivasan Janarthanam, Oliver Lemon, Xingkun Liu, Phil Bartie, William Mackaness, Tiphaine Dalmas and Jana Goetze, 2012, "Integrating location, visibility, and Question-Answering in a spoken dialogue system for Pedestrian City Exploration", Accepted at SIGDIAL 2012, South Korea.
  • Srinivasan Janarthanam and Oliver Lemon, 2012, "Influencing User Behaviour in Personalised Location Based Services", Accepted at Symposium on Influencing People using Information (SIPI 2012), Aberdeen.  Paper Poster Talk

Related project links: SPACEBOOK

First year project report  

Here is a demo of the first prototype on the streets of Edinburgh..


HELP4MOOD (since 2011)

@ Interaction Lab, HWU

Help4Mood aims at implementing and examining the effects of a virtual dialogue agent on patients with moderate depression. The dialogue agent will be designed to be a companion which administers standard questionnaires, suggests cognitive games, etc.

Related project links: HELP4MOOD


GRUVE (since 2011)

@ Interaction Lab, HWU

Generating Route Instructions under Uncertain Virtual Environments (GRUVE) is a generation challenge that invites participating teams to develop interaction manager and natural language generation modules for conversations involving navigation tasks in a virtual city-like environments.

Related publications:

  • Srinivasan Janarthanam and Oliver Lemon, 2012,  "A Web-based Evaluation Framework for Spatial Instruction-Giving Systems", Accepted at ACL conference 2012, South Korea. paper
  • Srini Janarthanam and Oliver Lemon, "The GRUVE Challenge: Generating Route Instructions under Uncertain Virtual Environments", GenChal workshop, ENLG 2011, Nancy, France. paper

Related talks:

  • Srini Janarthanam and Oliver Lemon, "The GRUVE Challenge: Generating Route Instructions under Uncertain Virtual Environments", UCNLG workshop at EMNLP conference, Edinburgh.


Temporal Expression Generation (2011)

School of Informatics, The University of Edinburgh, UK

We worked on learning temporal expression generation policies in the domain of appointment scheduling dialogues using reinforcement learning methods. Results show that temporal expression generation policy learned using RL techniques are better than hand-coded policies. Detailed results will be presented soon. CLASSiC project.

Related publications:

  • Srini Janarthanam, Helen Hastie, Oliver Lemon and Xingkun Liu, "'The day after the day after tomorrow?' A machine learning approach to adaptive temporal expression generation: training and evaluation with real users", In proceedings of SIGDial 2011, Portland, USA. link

Project Deliverable Reports:

  • Srini Janarthanam, Oliver Lemon, Romain Laroche, and Ghislain Putois, "Testing learned NLG and TTS policies with real users, in Self-Help and Appointment Scheduling Systems" link
  • Filip Jurcıcek, Simon Keizer, Francois Mairesse, Kai Yu, Steve Young, Srini Janarthanam, Helen Hastie, Xingkun Liu, and Oliver Lemon, "Proof-of-concept CLASSIC Appointment Scheduling system (System 2)" link

Related Project Links: 


Learning User Modelling Policies for Adaptive Referring Expression Generation in Spoken Dialogue Systems (2007-2010)

Ph.D thesis @ School of Informatics, The University of Edinburgh, UK

I investigated how to make machines to adapt to different users by speaking in a language that they would understand. In other words, how to learn user modeling strategies for dialogue management and natural language generation that adapt to users with different levels of expertise in Spoken Dialogue Systems.  In domains like Technical Support and City Navigation, the policies must adapt to different kinds of users like beginners, intermediates and experts. How to modify the dialogue state to track the user's domain knowledge? How to simulate different kinds of users? How to simulate user's referring expression recognition process? These are questions that we try to answer.

Using a reinforcement learning framework, we learn policies that can adapt to different kinds of users (e.g. beginners, intermediates, experts) in a domain and produce utterances that suit the user. See research publications for more details. We showed that adaptive systems built this way produced 99.47% successful task completion and approx. 11% reduction in dialogue duration in comparison to some hand-coded adaptive systems.

Related publications:

  • Srinivasan Janarthanam & Oliver Lemon. 2010d. Learning Adaptive Referring Expression Generation Policies for Spoken Dialogue Systems, In: Krahmer, E., Theune, M. (eds.) Empirical Methods in Natural Language Generation, LNCS, vol. 5980. Springer, Berlin / Heidelberg (2010). Springer site
  • Srinivasan Janarthanam and Oliver Lemon. 2010c. Adaptive Referring Expression Generation in Spoken Dialogue Systems: Evaluation with Real UsersAccepted. SigDial 2010 (Tokyo, Japan). paper
  • Srinivasan Janarthanam and Oliver Lemon. 2010b. Learning to Adapt to Unknown Users: Referring Expression Generation in Spoken Dialogue SystemsIn proc. ACL 2010 (Uppsala, Sweden)paper

  • Oliver Lemon, Srinivasan Janarthanam and Verena Rieser. 2010a. Generation under uncertainty. In proc. Sixth International  Natural Language Generation conference (INLG 2010), Dublin. paper

  • Srinivasan Janarthanam and Oliver Lemon. 2009e. A Two-tier User Simulation Model for Reinforcement Learning of Adaptive Referring Expression Generation PoliciesIn proc. SIGDial 2009 (London).paper

  • Srinivasan Janarthanam and Oliver Lemon. 2009d. A Data-driven method for Adaptive Referring Expression Generation in Automated Dialogue Systems: Maximising Expected UtilityIn proc. PRE-CogSci 2009 (Amsterdam, Netherlands)paper
  • Srinivasan Janarthanam and Oliver Lemon. 2009c. Learning Adaptive Referring Expression Generation Policies for Spoken Dialogue Systems using Reinforcement Learning. In proc. SEMDIAL 2009 (Stockholm, Sweden)paperposter
  • Srinivasan Janarthanam and Oliver Lemon. 2009b. A Wizard-of-Oz Environment to study Referring Expression Generation in a Situated Spoken Dialogue Task. In proc. ENLG-EACL 2009 (Athens, Greece).paper
  • Srinivasan Janarthanam and Oliver Lemon. 2009a. Learning Lexical Alignment Policies for Generating Referring Expressions for Spoken Dialogue Systems. In proc. ENLG-EACL 2009 (Athens, Greece). paper
  • Srinivasan Janarthanam and Oliver Lemon. 2008. User simulation for knowledge-alignment and online adaptation in Troubleshooting Dialogue Systems. In proc SEMDIAL 2008 (LONDIAL), London. paper

Project Deliverable Reports:

  • Oliver Lemon, Verena Rieser, Srini Janarthanam, Xingkun Liu, "Reinforcement Learning of optimal NLG policies using simulated users, for TownInfo and SelfHelp systems" link
  • Stephane Rossignol, Srini Janarthanam, Xingkun Liu, Olivier Pietquin, Michel Ianotto, "User simulations of different types of Appointment Scheduling and Self-Help user" link
  • Verena Rieser, Srinivasan Janarthanam, Oliver Lemon, "Simulated users for training NLG (TownInfo/Self-Help systems)" link
  • Cedric Boidin, Verena Rieser, Srinivasan Janarthanam, Oliver Lemon, "Domain-limited TTS corpus for expressive speech synthesis and Wizard-of-Oz Data for NLG Strategies" link

Related talks:

  • Multi-modal interaction in University of Edinburgh at SICSA Multi-modal Interaction workshop at University of St. Andrews on 11th Feb 2011.
  • Second Year Ph.D. review talk 2009 Talk 
  • Dialogue systems in Education at Center for Internet and Society, Bangalore on 12th June 2009Talkmore
  • User simulations for Self-help dialogue systems at IRTG 2008, University of Edinburgh
  • User simulations for learning adaptive dialogue policies at Informatics Jamboree 2008, University of Edinburgh
  • Tutorial dialogue systems at ELSNET 2007 (student presentations), Queen's University Belfast.

Related Project Links: 


Cross Language Information Retrieval - Transliterating English Named Entities to Tamil (2007)

Information is now available in multiple languages. Wouldn't it be easier if your search engine can fetch information from multi-lingual archives and present you with relevant documents in multiple languages, instead of forcing you to perform multiple searches once in every language. In order to provide such a service, the search engines must translate your English query in to multiple languages and search the respective archive. A major part of user's queries are named entities, like person names, city names, etc which cannot be translated. They must be transliterated, a process which involves transcribing the name in the target language with a matching pronunciation. In this paper, we present an algorithm to transliterate English named entities in to Tamil.

Related publication:

  • Srinivasan Janarthanam, Sethuramalingam S and Udhyakumar Nallasamy, Named Entity Transliteration for Cross Language Information Retrieval using Compressed Word Format Algorithm, 2nd International ACM Workshop Improving Non-English Web Searching (iNEWS-08), California.


NLP tools for Tamil (2005-2006)

Speech & NLP Group, CEN, AMRITA

Morphological Analyzer: We developed a Tamil Morphological Analyser that can analyse the given word, identify its root and its features, both semantic and syntactic and present a parse tree and a feature structure of the analysis. The analyser can be domain specific or open-domain depending on the lexicon size. We have used the analyser for our spoken dialog system already and we are now working towards an universal open-domain analyser for Tamil.

Dependency parser: Constructing phrase structure trees for a sentence seems to be particularly very difficult in Tamil. Tamil allows the various constituents to move around inside the clause they belong to. Hence positional analysis done by Phrase strucure CFGs seems inappropriate to Tamil. Hence we are developing Dependency framework for Tamil that will parse the given sentences and present a dependency analysis. The output will be a dependency tree instead of a phrase structure tree. We use our morphological analyser to provide the parser with the feature structures that will be used to identify the dependency relations.

Dialogue manager: Continuing my earlier work on building NL interfaces, I am working on Spoken Dialog Systems for Tamil. A lot of potential applications can come out of this technology, very useful especially for a developing country like India with multi-lingual population. This project aims to build a speech interfaced system that can handle queries on train enquiry, seat reservation, etc.

Related publication:

  • Srinivasan Janarthanam, Udhaykumar N, Loganathan R, Santhoshkumar C. 2007. Robust Dependency Parser for Natural Language Dialog Systems in Tamil. Proceedings of 5th Workshop on Knowledge and Reasoning in Practical Dialogue Systems in IJCAI-2007, Hyderabad, India. pdf


Optical Character Recognition for Tamil (2005-2006)

Speech & NLP Group, CEN, AMRITA
Tamil character set comprises of around 150 unique print characters. Using Multiclass Hierarchical Support Vector Machine classifiers we are developing an optical character recognition program for Tamil.

  • Shivsubramani K, Loganathan R, Srinivasan Janarthanam, Ajay V, Soman KP. Multiclass Hierarchical SVM for Recognition of Printed Tamil Characters. Proceedings of IJCAI-2007 Workshop on Analytics for Noisy Unstructured Text Data, IJCAI-2007, India pdf

Undergrad Student Projects supervision @ Amrita University

  • Text Summarization
  • Dialogue systems (Personal assistant, Medical diagnosis)
  • Machine Translation (English-Tamil & English-Hindi)


Learning Tamil Morphophonemic Rules using Inductive Logic Programming (2004-2005)

M.Sc Thesis @ University of Sussex, UK. 

Supervisor: Dr. Bill Keller, Lecturer in AI, U. Sussex.

Inductive Logic Programming is a machine learning strategy that learns prolog rules from positive and negative examples. Using Suresh Manadhar's CLOG learner, a system to learn morphophonemic rules of Tamil was devised. Data was collected from a leading newspaper and was hand annotated. Only positive examples were created. Using the Closed World Assumption, the incorrect theories were avoided. The system was capable of identifying morpheme boundaries and was able to perform morphophonemic mutations when identifying root words from inflected forms. The sytem was able to achieve an accuracy of around 84% in recognising morpheme boundaries and splitting them. Additionally, the system tried to PoS tag the analysed wordform. Using hand-annotated examples and CLOG, the system was able to disambiguate instances where multiple PoS tags were possible.


English - Tamil Machine Translation (2003)

@ Lingusitics Studies Unit, University of Madras.

Transfer method is one of the most commonly used strategies for Machine Translation. In this short-term project, we tried to develop a system that can translate simple constructions in English to Tamil. Transfer rules called Synmaps were created by analysing Tamil translations of English sentences. These rules map English constructions to Tamil at all levels of constructions, starting from words to phrases to clauses. English and Tamil have vary different syntax and morphology. These discrepancies are accommodated in the transfer rules. The system was able to successfully translate simple sentences from English to Tamil, for limited vocabulary. The system was a prototype and we didn't develop it to handle complex sentences.


Natural Language Interface for Library Database (1997-98)

Amrita Vishwa Vidyapeetham, Coimbatore

Undergraduate Thesis Supervisor: Dr. C. Karthikeyan

For my undergraduate project, I teamed up with three other people to build some application that displays Natural Language capabilities. We decided to build a Natural Language Interface for our library database. A scale-down version of the database was built using MS-ACCESS. We built a lexicon by collecting the vocabulory of the library domain. A detailed survey of dialogues between the user and library help-desk assistant was done. We listed down around 20 situations that can arise at the help-desk and asked our friends to write down how they will question the help-desk assistant in such situations. The dialogs were carefully studied and the grammar was created. The grammar consisted of CFG rules, which were in turn translated to semantic representation (FOPL). The dialog manager will read the semantic representation of the user's speech act and will translate that into an SQL query. This will be used to retrieve the data from our database. Our system had capabilities of resolving anaphora as well. References to books, authors and publishers in the current context can be made using appropriate pronouns during the discourse. Subjective evaluation was done by our friends, as they interacted with the system and tested how much of its response were appropriate. Our system achieved an accuracy of 85%.