Imagine placing an online order on your way to the grocery store, then being able to pick the collected basket upon arrival or shortly after. Likewise, imagine placing any online retail order, made ready for pickup in minutes instead of days. In order to realize such a low-latency automatic warehouse logistics system, solvers must be made to be basketaware. That is, it is more important that the full order (the basket) is picked timely and fast, than that any single item in the order is picked quickly. Current state-of-the-art methods are not basket-aware. Nor are they optimized for a positive customer experience, that is; to prioritize customers based on queue place and the difficulty associated with picking their order. An example of the latter is that it is preferable to prioritize a customer ordering a pack of diapers over a customer shopping a larger order, but only as long as the second customer has not already been waiting for too long. In this work we formalize the problem outlined, propose a new method that significantly outperforms the state-of-the-art, and present a new realistic simulated benchmark. The proposed method is demonstrated to work in an on-line and real-time setting, and to solve the on-demand multi-agent basket picking problem for automated shopping stores under realistic conditions.

Real‐world autonomous systems are expected to be increasingly deployed and operating in real‐world environments over the coming decades. Autonomous systems such as AI‐enabled robotic systems and intelligent transportation systems, will alleviate mundane human work, provide new services, and facilitate a smarter and more flexible infrastructure. The real‐ world environments affected include workplaces, public spaces, and homes. To ensure safe operations, in for example the vicinity of people, it is paramount that the autonomous systems are explainable, behave predictable, and can handle that the real world is ever changing and only partially observable. To deal with a dynamic and changing environment, consistently and safely, it is necessary to have sound uncertainty management. Explicit uncertainty quantification is fundamental to providing probabilistic safety guarantees that can also be monitored during runtime to ensure safety in new situations. It is further necessary for well‐grounded prediction and classification uncertainty, for achieving task effectiveness with high robustness and for dealing with unknown unknowns, such as world model divergence, using anomaly detection. This dissertation focuses on the notion of motion in terms of trajectories, from recognizing – to anticipating – to generating – to monitoring that it fulfills expectations such as predictability or other safety constraints during runtime. Efficiency, effectiveness, and safety are competing qualities, and in safety-critical applications the required degree of safety makes it very challenging to reach useful levels of efficiency and effectiveness. To this end, a holistic perspective on agent motion in complex and dynamic environments is investigated. This work leverage synergies in well‐founded formalized interactions and integration between learning, reasoning, and interaction, and demonstrate jointly efficient, effective, and safe capabilities for autonomous systems in safety‐critical situations.

Coverage Path Planning is the work horse of contemporary service task automation, powering autonomous floor cleaning robots and lawn mowers in households and office sites. While steady progress has been made on indoor cleaning and outdoor mowing, these environments are small and with simple geometry compared to general urban environments such as city parking garages, highway bridges or city crossings. To pave the way for autonomous road sweeping robots to operate in such difficult and complex environments, a benchmark suite with three large-scale 3D environments representative of this task is presented. On this benchmark we evaluate a new Coverage Path Planning method in comparison with previous well performing algorithms, and demonstrate state-of-the-art performance of the proposed method. Part of the success, for all evaluated algorithms, is the usage of automated domain adaptation by in-the-loop parameter optimization using Bayesian Optimization. Apart from improving the performance, tedious and bias-prone manual tuning is made obsolete, which makes the evaluation more robust and the results even stronger.

Lattice-based motion planning is a hybrid planning method where a plan is made up of discrete actions, while simultaneously also being a physically feasible trajectory. The planning takes both discrete and continuous aspects into account, for example action pre-conditions and collision-free action-duration in the configuration space. Safe motion planning rely on well-calibrated safety-margins for collision checking. The trajectory tracking controller must further be able to reliably execute the motions within this safety margin for the execution to be safe. In this work we are concerned with introspective learning and reasoning about controller performance over time. Normal controller execution of the different actions is learned using machine learning techniques with explicit uncertainty quantification, for safe usage in safety-critical applications. By increasing the model accuracy the safety margins can be reduced while maintaining the same safety as before. Reasoning takes place to both verify that the learned models stays safe and to improve collision checking effectiveness in the motion planner using more accurate execution predictions with a smaller safety margin. The presented approach allows for explicit awareness of controller performance under normal circumstances, and detection of incorrect performance in abnormal circumstances. Evaluation is made on the nonlinear dynamics of a quadcopter in 3D using simulation.

This paper presents preliminary work on using deep neural networksto guide general-purpose heuristic algorithms for performing utilitarian combinatorial assignment. In more detail, we use deep learning in an attempt to produce heuristics that can be used together with e.g., search algorithms to generatefeasible solutions of higher quality more quickly. Our results indicate that ourapproach could be a promising future method for constructing such heuristics.

Cyber-physical systems such as robots and intelligent transportation systems are heavy producers and consumers of trajectory data. Making sense of this data and putting it to good use is essential for such systems. When industrial robots, intelligent vehicles and aerial drones are intended to co-exist, side-by-side, with people in human-tailored environments safety is paramount. Safe operations require uncertainty-aware motion pattern recognition, incremental reasoning and rapid decision-making to manage collision avoidance, monitor movement execution and detect abnormal motion. We investigate models and techniques that can support and leverage the interplay between these various trajectory-based capabilities to improve the state-of-the-art for intelligent autonomous systems.

Robot safety is of growing concern given recent developments in intelligent autonomous systems. For complex agents operating in uncertain, complex and rapidly-changing environments it is difficult to guarantee safety without imposing unrealistic assumptions and restrictions. It is therefore necessary to complement traditional formal verification with monitoring of the running system after deployment. Runtime verification can be used to monitor that an agent behaves according to a formal specification. The specification can contain safety-related requirements and assumptions about the environment, environment-agent interactions and agent-agent interactions. A key problem is the uncertain and changing nature of the environment. This necessitates requirements on how probable a certain outcome is and on predictions of future states. We propose Probabilistic Signal Temporal Logic (ProbSTL) by extending Signal Temporal Logic with a sub-language to allow statements over probabilities, observations and predictions. We further introduce and prove the correctness of the incremental stream reasoning technique progression over well-formed formulas in ProbSTL. Experimental evaluations demonstrate the applicability and benefits of ProbSTL for robot safety.

Validation and verification of supervised machine learning models is becoming increasingly important as their complexity and range of applications grows. This paper describes an extension to Bayesian optimization which allows for selecting both training and validation data, in cases where data can be generated or calculated as a function of a spatial location.

Exploration is an important aspect of robotics, whether it is for mapping, rescue missions or path planning in an unknown environment. Frontier Exploration planning (FEP) and Receding Horizon Next-Best-View planning (RH-NBVP) are two different approaches with different strengths and weaknesses. FEP explores a large environment consisting of separate regions with ease, but is slow at reaching full exploration due to moving back and forth between regions. RH-NBVP shows great potential and efficiently explores individual regions, but has the disadvantage that it can get stuck in large environments not exploring all regions. In this work we present a method that combines both approaches, with FEP as a global exploration planner and RH-NBVP for local exploration. We also present techniques to estimate potential information gain faster, to cache previously estimated gains and to exploit these to efficiently estimate new queries.

A key requirement of autonomous vehicles is the capability to safely navigate in their environment. However, outside of controlled environments, safe navigation is a very difficult problem. In particular, the real-world often contains both complex 3D structure, and dynamic obstacles such as people or other vehicles. Dynamic obstacles are particularly challenging, as a principled solution requires planning trajectories with regard to both vehicle dynamics, and the motion of the obstacles. Additionally, the real-time requirements imposed by obstacle motion, coupled with real-world computational limitations, make classical optimality and completeness guarantees difficult to satisfy. We present a unified optimization-based motion planning and control solution, that can navigate in the presence of both static and dynamic obstacles. By combining optimal and receding-horizon control, with temporal multi-resolution lattices, we can precompute optimal motion primitives, and allow real-time planning of physically-feasible trajectories in complex environments with dynamic obstacles. We demonstrate the framework by solving difficult indoor 3D quadcopter navigation scenarios, where it is necessary to plan in time. Including waiting on, and taking detours around, the motions of other people and quadcopters.

Conventional trajectory-based vehicular traffic analysis approaches work well in simple environments such as a single crossing but they do not scale to more structurally complex environments such as networks of interconnected crossings (e.g. urban road networks). Local trajectory models are necessary to cope with the multi-modality of such structures, which in turn introduces new challenges. These larger and more complex environments increase the occurrences of non-consistent lack of motion and self-overlaps in observed trajectories which impose further challenges. In this paper we consider the problem of motion pattern recognition in the setting of sequential local motion pattern models. That is, classifying sub-trajectories from observed trajectories in accordance with which motion pattern that best explains it. We introduce a Gaussian process (GP) based modeling approach which outperforms the state-of-the-art GP based motion pattern approaches at this task. We investigate the impact of varying local model overlap and the length of the observed trajectory trace on the classification quality. We further show that introducing a pre-processing step filtering out stops from the training data significantly improves the classification performance. The approach is evaluated using real GPS position data from city buses driving in urban areas for extended periods of time.

With Twitter and other microblogging services, users can easily express their opinion and ideas in short text messages. A recent trend is that users use the real-time property of these services to share their opinions and thoughts as events unfold on TV or in the real world. In the context of TV broadcasts, Twitter (over a mobile device, for example) is referred to as a second screen. This paper presents the first characterization of the second screen usage over the playoffs of a major sports league. We present both temporal and spatial analysis of the Twitter usage during the end of the National Hockey League (NHL) regular season and the 2015 Stanley Cup playoffs. Our analysis provides insights into the usage patterns over the full 72-day period and with regards to in-game events such as goals, but also with regards to geographic biases. Quantifying these biases and the significance of specific events, we then discuss and provide insights into how the playoff dynamics may impact advertisers and third-party developers that try to provide increased personalization.

Integrating logical and probabilistic reasoning and integrating reasoning over observations and predictions are two important challenges in AI. In this paper we propose P-MTL as an extension to Metric Temporal Logic supporting temporal logical reasoning over probabilistic and predicted states. The contributions are (1) reasoning over uncertain states at single time points, (2) reasoning over uncertain states between time points, (3) reasoning over uncertain predictions of future and past states and (4) a computational environment formalism that ground the uncertainty in observations of the physical world. Concrete robot soccer examples are given.

Integrating logical and probabilistic reasoning and integrating reasoning over observations and predictions are two important challenges in AI. In this paper we propose P-MTL as an extension to Metric Temporal Logic supporting temporal logical reasoning over probabilistic and predicted states. The contributions are (1) reasoning over uncertain states at single time points, (2) reasoning over uncertain states between time points, (3) reasoning over uncertain predictions of future and past states and (4) a computational environment formalism that ground the uncertainty in observations of the physical world. Concrete robot soccer examples are given.

Learning to recognize common activities such as traffic activities and robot behavior is an important and challenging problem related both to AI and robotics. We propose an unsupervised approach that takes streams of observations of objects as input and learns a probabilistic representation of the observed spatio-temporal activities and their causal relations. The dynamics of the activities are modeled using sparse Gaussian processes and their causal relations using a probabilistic graph. The learned model supports in limited form both estimating the most likely current activity and predicting the most likely future activities. The framework is evaluated by learning activities in a simulated traffic monitoring application and by learning the flight patterns of an autonomous quadcopter.

Trajectories are used in many target tracking and other fusion-related applications. In this paper we consider the problem of modeling trajectories as Gaussian processes and learning such models from sets of observed trajectories. We demonstrate that the traditional approach to Gaussian process regression is not suitable when modeling a set of trajectories. Instead we introduce an approach to Gaussian process trajectory regression based on an alternative way of combing two Gaussian process (GP) trajectory models and inverse GP regression. The benefit of our approach is that it works well online and efficiently supports sophisticated trajectory model manipulations such as merging and splitting of trajectory models. Splitting and merging is very useful in spatio-temporal activity modeling and learning where trajectory models are considered discrete objects. The presented method and accompanying approximation algorithm have time and memory complexities comparable to state of the art of regular full and approximative GP regression, while havinga more flexible model suitable for modeling trajectories. The novelty of our approach is in the very flexible and accurate model, especially for trajectories, and the proposed approximative method based on solving the inverse problem of Gaussian process regression.

Learning to recognize and predict common activities, performed by objects and observed by sensors, is an important and challenging problem related both to artificial intelligence and robotics.In this thesis, the general problem of dynamic adaptive situation awareness is considered and we argue for the need for an on-line bottom-up approach.A candidate for a bottom layer is proposed, which we consider to be capable of future extensions that can bring us closer towards the goal.We present a novel approach to adaptive activity learning, where a mapping between raw data and primitive activity concepts are learned and continuously improved on-line and unsupervised. The approach takes streams of observations of objects as input and learns a probabilistic representation of both the observed spatio-temporal activities and their causal relations. The dynamics of the activities are modeled using sparse Gaussian processes and their causal relations using probabilistic graphs.The learned model supports both estimating the most likely activity and predicting the most likely future (and past) activities. Methods and ideas from a wide range of previous work are combined to provide a uniform and efficient way to handle a variety of common problems related to learning, classifying and predicting activities.The framework is evaluated both by learning activities in a simulated traffic monitoring application and by learning the flight patterns of an internally developed autonomous quadcopter system. The conclusion is that our framework is capable of learning the observed activities in real-time with good accuracy.We see this work as a step towards unsupervised learning of activities for robotic systems to adapt to new circumstances autonomously and to learn new activities on the fly that can be detected and predicted immediately.

We propose an unsupervised stream processing framework that learns a Bayesian representation of observed spatio-temporal activities and their causal relations. The dynamics of the activities are modeled using sparse Gaussian processes and their causal relations using a causal Bayesian graph. This allows the model to be efficient through compactness and sparsity in the causal graph, and to provide probabilities at any level of abstraction for activities or chains of activities. Methods and ideas from a wide range of previous work are combined and interact to provide a uniform way to tackle a variety of common problems related to learning, classifying and predicting activities. We discuss how to use this framework to perform prediction of future activities and to generate events.

For people with hearing impairments, attending to a single speaker in a multi-talker background can be very difficult and something which the current hearing aids can barely help with. Recent studies have shown that the audio stream a human focuses on can be found among the surrounding audio streams, using EEG and linear models. With this rises the possibility of using EEG to unconsciously control future hearing aids such that the attuned sounds get enhanced, while the rest are damped. For such hearing aids to be useful for every day usage it better be using something other than a motion sensitive, precisely placed EEG cap. This could possibly be archived by placing the electrodes together with the hearing aid in the ear.One of the leading hearing aid manufacturer Oticon and its research lab Erikholm Research Center have recorded an EEG data set of people listening to sentences and in which electrodes were placed in and closely around the ears. We have analyzed the data set by applying a range of signal processing approaches, mainly in the context of audio estimation from EEG. Two different types of linear sparse models based on L1-regularized least squares are formulated and evaluated, providing automatic dimensionality reduction in that they significantly reduce the number of channels needed. The first model is based on linear combinations of spectrograms and the second is based on linear temporal filtering. We have investigated the usefulness of the in-ear electrodes and found some positive indications. All models explored consider the in-ear electrodes to be the most important, or among the more important, of the 128 electrodes in the EEG cap.This could be a positive indication of the future possibility of using only electrodes in the ears for future hearing aids.