Multimodal Interaction for Information Appliances

Project objectives

• Multimodal interaction
• Information Processing

Project work plan

Project prototypes

Project Summary

In the future e-Home, information from various sources, located both globally and locally, are at hand when performing a wide range of tasks. Some of these are mundane everyday tasks like cooking and cleaning the dishes. But a large number of them are performed on a less regular basis, sometimes only once a year or even more rarely. Examples of these are e.g. re-planting the flowers, or treating them for diseases, fixing the car, changing tires or mending a puncture on the bike, gardening tasks, etc. In the not too distant future we will probably include maintenance of the LAN, and installing new appliances in the family's network. Another set of tasks relates to the management of information for the household and its members. Major types of these are finding out about the public authorities' rules and regulations, e.g. rules for tax-deduction, the local area development plan, building permits, etc.

Imagine for instance a family member fixing the car, finding that a certain item is different from what is described in the manual. A quick request for information on what to do, maybe from the manufacturers information sources, will tell that the specific car is equipped with a different part. This will be presented in a convenient way depending on available interaction modalities. For such a problem-solving task it is of vital importance to be able to interact in an intuitive and efficient manner. We need to allow for multimodal interaction, i.e. the use and integration of various interaction modes including speech, graphics, images, videos, and text. Multimodal interaction requires usability investigations and the development of components for various purposes, including information extraction and sophisticated problem solving. Other important tasks concern manipulating or controlling information appliances in the home, for example, navigating through various entertainment channels using speech. With the increasing European integration, multi-lingual information issues must also be addressed.

The above home tasks differ in many aspects from tasks in professional situations, where most work is done by specialists. Professionals have to search for information, especially about details of the rules of procedures, but then with the difference that professional tasks are performed regularly.

In this project we will work on investigating and developing multimodal interaction systems of the kinds presented above. The long-term vision is to integrate such systems within a common e-home framework, but we believe that a fruitful research strategy will start from the specific examples, and work towards a common framework, instead of trying to develop such a framework in a top-down fashion.

The project is a joint project between the NLP group at Department of Computer and Information Science, Linköping University, Nokia, and IDA Systems. The project is initially organised around two subprojects, which together form the basis for multimodal dialogue with information appliances in the future e-home. Each subproject is focused on a specific research topic with a senior researcher in charge and, in most cases, a doctoral student employed. As stated above, the long-term vision is a single system, integrating several subcomponents.

The first year will involve investigations of the whole area and development of two prototypes, demonstrating multimodal dialogue for information appliances. One demonstrator will take its applications form the co-operation with Nokia, where language technology is applied for finding information on TV programs. This is closely related to ongoing research projects on using the Media Terminal at Nokia.<

The other demonstrator intends to show how language technology can enhance document management systems and communication between public agencies and the family. This subproject will be carried out in co-operation with IDA Systems, one of Sweden's main actors on the document management arena, with focus on information systems for public agencies.