Multimodal Interaction for Information Services

During the project, we will adopt the incremental development strategy. This means that the dialogue system will continuously be more and more advanced as we iteratively add capabilities.

Investigations on multimodal interaction

However, little is known of how the user experiences multimodal interaction. In the area of Human-Computer Interaction, researchers have been sceptical about using speech and or natural language for interaction with computers. Most arguments are based on the fact that the user loses control when using interaction techniques based on natural language. When using e.g. speech as interaction technique, the system makes a statistical interpretation of the users input, thus, interpretation of user input, and hence system response, can vary. This inconsistency means that the user never knows how the system will respond. A more philosophical argument for loss of control in multimodal interaction is based on the fact that natural language requests are less direct than direct manipulation. A third argument is that a speech or multimodal system can give an impression of understanding more than it actually does, and it is not evident what the system really understands. A consequence of this is again a loss of control, since the user often do not know how to adopt to what the system understands. Even without previous misunderstandings from the system, users often find it hard to know what to say to a system.

The most common objection to these arguments are that humans use speech and multimodal interaction everyday, it is their natural way of communication. Furthermore, studies show that multimodal interaction is in fact more efficient.

In this sub-project the aim is to investigate what properties that are important for an acceptance of multimodal interaction from the users. Is control an important property of a multimodal system? If so, is control manifested in use of interaction techniques or in dialogue strategies? Is it important to be able to freely choose what interaction technique to use at any point in time, or to use several simultaneously, or restrict the use of interaction techniques in various points in the dialogue? How should the dialogue between the user and system be designed to maintain a sense of control in the user's view; Should it be very open, and unrestricted, or clearly restricted to certain input from the user?

Development and maintenance of shared knowledge sources

The lexicon is the main knowledge sources for the Interpreter in our framework. Fortunately, much of the information needed to create the lexicon can be acquired automatically by information extraction from the documents. Furthermore, if the grammar is based on partial information, as is the case in our framework, many auxiliary words are not needed, which also makes automatic extraction and lexicon development easier. Ontologies have a less clearly defined role. In our view, an ontology is a knowledge base with information about concepts, their properties and relations. In an open domain with unstructured information a number of research issues concerning ontologies must be addressed.

The first question to address is how the ontology is to be created. To hand-craft ontologies is a cumbersome and time consuming task, in an open and unstructured domain it can even be an unmanageable task. Techniques for automatic construction of ontologies are therefore highly prioritised issue to investigate. One possible approach is to use existing resources like thesauri, dictionaries, and taxonomies. A thesaurus or taxonomy provides a hierarchical structure, which can be extended with semantic relations derived from a dictionary. One such resource for Swedish could be SWordNet, the Swedish version of WordNet.

Another approach to automatic construction of ontologies is to identify and extract concepts and relations from text corpora. Different IE techniques and machine learning can be used. Although, not much is done here, there are promising results, for instance utilising genetic programming to discover hyponomy relations between concepts.

Once an ontology has been constructed it has to be maintained. Since new information can be added the ontology must be able to change accordingly. This implies that techniques for automatic updates of an ontology has to be developed.

As the ontology will be a central resource in the system and will be shared by various modules, the format and properties must serve different purposes. To discover these an investigation of how the domain ontology can be used by the different modules in a dialogue system has to be conducted.

Information extraction and document processing

Information sources are often in several languages and it will therefore be necessary for future information systems to handle multilingual information. Multilinguality is not a primary objective in this project, but the topic is still of natural interest to the group as two of the project members also are involved in projects concerning machine translation and multilingual processing within the Natural Language Laboratory at the department.

Furthermore, there is also an interest to investigate how ontology-based document processing will promote advances within automatic summarisation. These research areas are indeed connected and we plan to do some preliminary investigations in this direction.

In this subproject, emphasis will be put on developing a wide set of semantic wrappers (both general and domain-specific) that will be tested and evaluated on several Swedish information domains. Naturally, this is tightly connected to the ontologies and how these are used as a common resource for several components of the project.

Dialogue Systems development

The MALIN interpreter will be refined with a capable component for query analysis. This will decrease the complexity of formulating information requests by allowing intuitive questions in natural language. To be able to correctly decipher the intended meaning of a user question is one of the critical issues in a Q&A system. Vague, ambiguous and partial input has to be dealt with. By using domain knowledge from the ontology together with syntactic surface analysis, as well as dialogue history information, a catalogue of question categories on different levels will be developed. The selected question category for a particular query will then form the basis for the query that is fed into the extraction module. Furthermore, the choice of question type gives the system information about how the answer should be presented to the user.

The central component in the MALIN dialogue system framework is the Dialogue Manager (DM) that maintains the dialogue history and handles the interaction with the user, i.e. control the flow of the dialogue by deciding how to respond to user utterances. In many information-providing dialogue systems the Dialogue Manager has also been responsible for the retrieval of requested information from the background systems. As a consequence, the dialogue, task and domain knowledge and reasoning has often been integrated. This integration has a number of drawbacks, for example, it makes it hard and time consuming to port the system to a new domain, a separation of domain from dialogue and task knowledge and reasoning has therefore been proposed.

In the MALIN dialogue system framework a separate module called the Domain Knowledge Manager (DKM) has been introduced. The DKM is responsible for domain reasoning and retrieval of information from various domain knowledge sources. The DKM cooperates with the DM to answer questions for information posed by the user. User utterances are transformed into information requests by the DM, possibly involving clarification subdialogues with the user. The fully specified request is then sent to the DKM. The DKM consults one or several information or domain knowledge sources in order to retrieve the requested information and produces an answer to the request. This requires that the DKM knows where and how different types of information should be retrieved, a task that becomes difficult when the domain is open and the information unstructured.

The introduction of the separate module for domain knowledge management makes the MALIN framework suitable for development of dialogue systems in unstructured and open domains. The DKM can act as an intermediator between the two subareas of information-processing and multi-modal interaction. However, the DM and DKM were developed for closed and structured domains. Investigations on how they can be extended to work on partially structured documents is one important research issue.

For dialogue management this involves adaptation of the dialogue and task models used by the dialogue manager to handle the dialogue. It also includes issues on how to utilise the domain ontology.