Multimodal Interaction for Information Appliances
Project prototypesThe project has produced two prototypes, one on text extraction and one on dialogue systems development.
Ask RSVThe prototype system Ask RSV utilises text extraction techniques to be able to answer questions in natural language in the domain of the Swedish tax area. In the system a general representation of tax concepts (a domain ontology) is used by wrappers to provide structure to unstructured documents in the domain. The result is an enriched set of documents that include semantic information and serves as the knowledge base for a questioning and answering interface. Sample interactions are shown in the following two figures:
The Nokia TV guideIn this project we utilised a framework for dialogue systems development, MALIN, that was customised for TV program information to be used in the Nokia Media Terminal. The dialogue system was implemented iteratively based on a method developed in the project, see the list of scientific reports.
The prototype was in the first iteration only able to respond to isolated questions, i.e. no context could be utilised when interpreting a request. This was then refined, in the second iteraion, to include also a history list, which meant that utterances could be interpreted in context. The third iteration allowed also for clarification requests, e.g.:
The fourth iteration added background information. Screenshot after asking What is Gladiator about
The fifth iteration added the ability to remember also the list of objects that was most recent, e.g. again refeering to all the films but restricting the request differently uttering Movies after 9 pm in the context before.
The latest version allows spoken interaction and can be seen in a QuickTime movie.
Page responsible: Webmaster
Last updated: 2012-05-07