Multimodal Interaction for Information Services

The overall research goal of this proposal is to incorporate document and text processing techniques in multimodal dialogue systems. The main result is, thus, a natural language information system, where multimodal interaction allows for intuitive and efficient formulation of complex requests for information that can be extracted from unstructured, distributed information sources.

Such an integrated multimodal dialogue system requires a variety of shared knowledge sources, especially important is a common ontology, that can be utilised by both the dialogue system and the information extraction components. The development of a general ontology is far beyond the research goals of this project. However, we expect to have ontologies that are useful for various types of applications and domains. Furthermore, one research issue that will be investigated concerns maintaining ontologies in a dynamic environment.

Another important goal is to understand multimodal human computer interaction, and especially issues related to control and co-operation. The expected results are on the one hand knowledge on multimodal interaction, but also principles for design of multimodal dialogue systems.