Human communication is multimodal in nature, and occurs through combinations of speech, language, gesture, facial expression, and similar signals. To enable natural interactions with human beings, artificial agents must be capable of both analysing and producing these rich and interdependent signals, and connect them to their semantic implications. Unfortunately, even the strongest machine learning methods currently fall short of this goal: automated semantic understanding of human behaviour remains superficial, and generated agent behaviours are empty gestures lacking the ability to convey meaning and communicative intent.

The STING NEST intends to change this state of affairs by uniting synthesis and analysis with transducers and invertible neural models. This involves connecting concrete, continuous­valued sensory data such as images, sound, and motion, with high­level, predominantly discrete, representations of meaning, which has the potential to endow synthesis output with human­understandable high­level explanations, while simultaneously improving the ability to attach probabilities to semantic representations. The bi­directionality also allows us to create efficient mechanisms for explainability, and to inspect and enforce fairness in the models.

Recent advances in generative models suggest that our ambitious research agenda is likely to be met with success. Normalising flows and variational autoencoders permit both extracting disentangled representations of observations, and (re­)generating observations from these abstract representations, all within a single model. Their recent extensions to graph­structured data are of particular interest because graphs are commonly­used semantic representations. This opens the door not only to generating structured information, but also to capturing the composition of the generation itself (which is a graph in its own right) by exploiting and transferring techniques from finite­state transducers and graph grammars.

In addition to its scientific value, the project is expected to have a substantial societal imprint. The resulting technologies may, e.g., be used to create virtual patients for medical training, to model non­playable characters in video games, and to derive affective states and underlying health issues from human speech and non­verbal behaviour.

• PI: Gustav Eje Henter (KTH)
• Co-PIs: Henrik Björklund, Johanna Björklund, Frank Drewes (all Umeå University), Hedvig Kjellström (KTH), Marco Kuhlmann
• Project staff: tbd
• Funding from: WASP (Wallenberg AI, Autonomous Systems and Software Program)
• Project period: 2022–2026

Neural networks are today the most important computational paradigm for AI and machine learning. While their computational power has been studied for decades, the theoretical development has not been able to keep up with the recent upshot of new neural architectures, such as the Transformer. Also, existing theoretical results are of limited practical value because they often apply to monolithic architectures, whereas applications typically use networks that are composed of re-usable components. Moreover, theoretical results tend to make idealising assumptions such as infinite precision or unlimited training time, rather than acknowledging that such commodities are in short supply and should thus be treated as parameters of expressive power. In this project, we want to address these shortcomings and develop a practical theory of computation for modern neural network architectures. The project combines methods from theoretical computer science – especially the theory of computation and formal languages – with empirical validation in natural language processing (NLP).

• PIs: Marco Kuhlmann and Frank Drewes (Umeå University)
• Project staff: Lena Strobl, Olle Torstensson
• Funding from: WASP (Wallenberg AI, Autonomous Systems and Software Program)
• Project period: 2022–2026

The field of natural language processing (NLP) has seen major progress during the last few years with the development of deep neural language models, which learn tasks such as question answering, machine translation, and text summarization without any explicit supervision. This project will apply these models to the task of extracting semantic relations between named entities from raw text. Our main goal is to design, implement, and evaluate an end-to-end system for relation extraction based on deep neural language models. Because training these models from scratch is extremely resource-intensive, we are specifically interested in developing methods for maximizing the performance that can be obtained by fine-tuning pre-trained models, and in particular models for smaller languages such as Swedish.

• PI: Marco Kuhlmann
• Project staff: Ali Basirat
• Funding from: ELLIIT (Excellence Center at Linköping–Lund in Information Technology)
• Project period: 2021–2022

Building computers that understand human language is one of the central goals in artificial intelligence. A recent breakthrough on the way towards this goal is the development of neural models that learn deep contextualized representations of language. However, while these models have substantially advanced the state of the art in natural language processing (NLP) for a wide range of tasks, our understanding of the learned representations and our repertoire of techniques for integrating them with other knowledge representations and reasoning facilities remain severely limited. To address these gaps, we will develop new methods for the interpretation, grounding, and integration of deep contextualized representations of language, and to evaluate the usefulness of these methods in the context of threat intelligence applications together with our industrial partner, Recorded Future.

• PIs: Marco Kuhlmann and Richard Johansson (Chalmers University of Technology)
• Project staff: Ehsan Doostmohammadi, Lovisa Hagström, Tobias Norlund
• Funding from: WASP (Wallenberg AI, Autonomous Systems and Software Program)
• Project period: 2020–2025