Multimodal Interaction for Information Appliances


Executive Summary

Project objectives

Industrial relevance

Project work plan

Scientific production

Project prototypes



Project prototypes

The project has produced two prototypes, one on text extraction and one on dialogue systems development.


The prototype system Ask RSV utilises text extraction techniques to be able to answer questions in natural language in the domain of the Swedish tax area. In the system a general representation of tax concepts (a domain ontology) is used by wrappers to provide structure to unstructured documents in the domain. The result is an enriched set of documents that include semantic information and serves as the knowledge base for a questioning and answering interface.  Sample interactions are shown in the following two figures:
  • Ask RSV 1 (In Swedish)
  • Ask RSV 2 (In Swedish)

    The Nokia TV guide

    In this project we utilised a framework for dialogue systems development, MALIN, that was customised for TV program information to be used in the Nokia Media Terminal. The dialogue system was implemented iteratively based on a method developed in the project, see the list of scientific reports.

    The prototype was in the first iteration only able to respond to isolated questions, i.e. no context could be utilised when interpreting a request. This was then refined, in the second iteraion, to include also a history list, which meant that utterances could be interpreted in context. The third iteration allowed also for clarification requests, e.g.:

    Understanding that the rquest List all movies today provides to many alternatives and therefore asking for a specification. For instance, responding to the clarification request with No only tonight

    The fourth iteration added background information. Screenshot after asking What is Gladiator about

    The fifth iteration added the ability to remember also the list of objects that was most recent, e.g. again refeering to all the films but restricting the request differently uttering Movies after 9 pm in the context before.

    The latest version allows spoken interaction and can be seen in a QuickTime movie.

  • Page responsible: Webmaster
    Last updated: 2012-05-07