In the context of collaborative robotics, distributed situation awareness is essential for supporting collective intelligence in teams of robots and human agents where it can be used for both individual and collective decision support. This is particularly important in applications pertaining to emergency rescue and crisis management. During operational missions, data and knowledge is gathered incrementally and in different ways by heterogeneous robots and humans. The purpose of this paper is to describe an RDF Graph Synchronization System called RGS⊕. It is assumed that a dynamic set of agents provide or retrieve knowledge stored in their local RDF Graphs which are continuously synchronized between agents. The RGS⊕ System was designed to handle unreliable communication and does not rely on a static centralized infrastructure. It is capable of synchronizing knowledge as timely as possible and allows agents to access knowledge while it is incrementally acquired. A deeper empirical analysis of the RGS⊕ System is provided that shows both its efficiency and efficacy.

In the context of collaborative robotics, distributed situation awareness is essential for supporting collective intelligence in teams of robots and human agents where it can be used for both individual and collective decision support. This is particularly important in applications pertaining to emergency rescue and crisis management. During operational missions, data and knowledge are gathered incrementally and in different ways by heterogeneous robots and humans. We describe this as the creation of Hastily Formed Knowledge Networks (HFKNs). The focus of this paper is the specification and prototyping of a general distributed system architecture that supports the creation of HFKNs by teams of robots and humans. The information collected ranges from low-level sensor data to high-level semantic knowledge, the latter represented in part as RDF Graphs. The framework includes a synchronization protocol and associated algorithms that allow for the automatic distribution and sharing of data and knowledge between agents. This is done through the distributed synchronization of RDF Graphs shared between agents. High-level semantic queries specified in SPARQL can be used by robots and humans alike to acquire both knowledge and data content from team members. The system is empirically validated and complexity results of the proposed algorithms are provided. Additionally, a field robotics case study is described, where a 3D mapping mission has been executed using several UAVs in a collaborative emergency rescue scenario while using the full HFKN Framework.

Dealing with compressed video streams in mobile robotics is an unavoidable fact of life. Transferring images between mobile robots or to the Cloud using wireless links can practically only be achieved using lossy video compression. This introduces artifacts that often make image processing challenging. Recent algorithms based on deep neural networks, as advanced as they are, are commonly trained and evaluated on datasets of high-fidelity images which are typically not captured from aerial views. In this work we evaluate a number of deep neural network based object detection algorithms in the context of aerial search and rescue scenarios where real-time and robust detection of human bodies is a priority. We provide an evaluation using a number of video sequences collected in-flight using Unmanned Aerial Vehicle (UAV) platforms in different environmental conditions. We also describe the detection performance degradation under limited bitrate compression using H.264, H.265 and VP9 video codecs, in addition to analyzing the timing effects of moving image processing tasks to off-board entities.

Due to the maturity of technological development, widespread use of Unmanned Aerial Vehicles (UAVs) is becoming prevalent in the civil and commercial sectors. One promising area of application is in emergency rescue support. As recently seen in a number of natural catastrophes such as the hurricanes in Texas, Florida and Puerto Rico, major communication and electrical infrastructure is knocked out, leading to an inability to communicate between the victims and rescuers on the ground as well as between rescuers themselves. This paper studies the feasibility of using heterogeneous teams of UAVs to rapidly deliver and establish ad hoc communication networks in operational environments through autonomous in-air delivery of CommKits that serve as nodes in local ad hoc networks. Hardware and software infrastructures for autonomous CommKit delivery in addition to CommKit specification and construction is considered. The results of initial evaluation of two design alternatives for CommKits are presented based on more than 25 real flight tests in different weather conditions using a commercial small-scale UAV platform.

In this paper we address the issue of connecting abstract task definitions at a mission level with control functionalities for the purpose of performing autonomous robotic missions using multiple heterogenous platforms. The heterogeneity is handled by the use of a common vocabulary which consists of parametrized tasks such as fly-to, take-off, scan-area, or land. Each of the platforms participating in a mission supports a subset of the tasks by providing their platform-specific implementations. This paper presents a detailed description of an approach for implementing such platform-specific tasks. It is achieved using a flight-command based interface with setpoint generation abstraction layer for vertical take-off and landing platforms. We show that by using this highly expressive and easily parametrizable way of specifying and executing flight behaviors it is straightforward to implement a wide range of tasks. We describe the method in the context of a previously described robotics architecture which includes mission delegation and execution system based on a task specification language. We present results of an experimental flight using the proposed method.

This paper demonstrates how a heterogeneous fleet of unmanned aerial vehicles (UAVs) can support human operators in search and rescue (SaR) scenarios. We describe a fully autonomous delegation framework that interprets the top-level commands of the rescue team and converts them into actions of the UAVs. In particular, the UAVs are requested to autonomously scan a search area and to provide the operator with a consistent georeferenced 3D reconstruction of the environment to increase the environmental awareness and to support critical decision-making. The mission is executed based on the individual platform and sensor capabilities of rotary-and fixed-wing UAVs (RW-UAV and FW-UAV respectively): With the aid of an optical camera, the FW-UAV can generate a sparse point-cloud of a large area in a short amount of time. A LiDAR mounted on the autonomous helicopter is used to refine the visual point-cloud by generating denser point-clouds of specific areas of interest. In this context, we evaluate the performance of point-cloud registration methods to align two maps that were obtained by different sensors. In our validation, we compare classical point-cloud alignment methods to a novel probabilistic data association approach that specifically takes the individual point-cloud densities into consideration.

This paper presents the design and development of the LinkBoard, an advanced flight control system for micro Unmanned Aerial Vehicles (UAVs). Both hardware and software architectures are presented. The LinkBoard includes four processing units and a full inertial measurement unit. In the basic configuration, the software architecture includes a fully configurable set of control modes and sensor fusion algorithms for autonomous UAV operation. The system proposed allows for easy integration with new platforms, additional external sensors and a flexibility to trade off computational power, weight and power consumption. Due to the available onboard computational power, it has been used for computationally demanding applications such as the implementation of an autonomous indoor vision-based navigation system with all computations performed onboard. The autopilot has been manufactured and deployed on multiple UAVs. Examples of UAV systems built with the LinkBoard and their applications are presented, as well as an in-flight experimental performance evaluation of a newly developed attitude estimation filter.

Visual object tracking performance has improved significantly in recent years. Most trackers are based on either of two paradigms: online learning of an appearance model or the use of a pre-trained object detector. Methods based on online learning provide high accuracy, but are prone to model drift. The model drift occurs when the tracker fails to correctly estimate the tracked object’s position. Methods based on a detector on the other hand typically have good long-term robustness, but reduced accuracy compared to online methods.Despite the complementarity of the aforementioned approaches, the problem of fusing them into a single framework is largely unexplored. In this paper, we propose a novel fusion between an online tracker and a pre-trained detector for tracking humans from a UAV. The system operates at real-time on a UAV platform. In addition we present a novel dataset for long-term tracking in a UAV setting, that includes scenarios that are typically not well represented in standard visual tracking datasets.

The use of UAVs, in particular, micro VTOL UAVs, is becoming prevalent in emergency rescue and security applications, among others. In these applications, the platforms are tightly coupled to the human users and these applications require great flexibility in the interaction between the platforms and such users. During operation, one continually switches between manual, semi-autonomous and autonomous operation, often re-parameterising, breaking in, pausing, and resuming missions. One is in continual need of modifying existing elementary actions and behaviours such as FlyTo and TrackObject, and seamlessly switching between such operations. This paper proposes a flight command and setpoint abstraction layer that serves as an interface between continuous control and higher level elementary flight actions and behaviours. Introduction of such a layer into an architecture offers a versatile and flexible means of defining flight behaviours and dynamically parameterising them in the field, in particular where human users are involved. The system proposed is implemented in prototype and the paper provides experimental validation of the use and need for such abstractions in system architectures.

In this work we are investigating reactive avoidance techniques which can be used on board of a small quadcopter and which do not require absolute localisation. We propose a local map representation which can be updated with proprioceptive sensors. The local map is centred around the robot and uses spherical coordinates to represent a point cloud. The local map is updated using a depth sensor, the Inertial Measurement Unit and a registration algorithm. We propose an extension of the Dynamic Window Approach to compute a velocity vector based on the current local map. We propose to use an OctoMap structure to compute a 2-pass A* which provide a path which is converted to a velocity vector. Both approaches are reactive as they only make use of local information. The algorithms were evaluated in a simulator which offers a realistic environment, both in terms of control and sensors. The results obtained were also validated by running the algorithms on a real platform.

This paper describes an overview of a generic framework for collaboration among humans and multiple heterogeneous robotic systems based on the use of a formal characterization of delegation as a speech act. The system used contains a complex set of integrated software modules that include delegation managers for each platform, a task specification language for characterizing distributed tasks, a task planner, a multi-agent scan trajectory generation and region partitioning module, and a system infrastructure used to distributively instantiate any number of robotic systems and user interfaces in a collaborative team. The application focusses on 3D reconstruction in alpine environments intended to be used by alpine rescue teams. Two complex UAV systems used in the experiments are described. A fully autonomous collaborative mission executed in the Italian Alps using the framework is also described.

Robots are increasingly expected to move out of the controlled environment of research labs and into populated streets and workplaces. Collision avoidance in such cluttered and dynamic environments is of increasing importance as robots gain more autonomy. However, efficient avoidance is fundamentally difficult since computing safe trajectories may require considering both dynamics and uncertainty. While heuristics are often used in practice, we take a holistic stochastic trajectory optimization perspective that merges both collision avoidance and control. We examine dynamic obstacles moving without prior coordination, like pedestrians or vehicles. We find that common stochastic simplifications lead to poor approximations when obstacle behavior is difficult to predict. We instead compute efficient approximations by drawing upon techniques from machine learning. We propose to combine policy search with model-predictive control. This allows us to use recent fast constrained model-predictive control solvers, while gaining the stochastic properties of policy-based methods. We exploit recent advances in Bayesian optimization to efficiently solve the resulting probabilistically-constrained policy optimization problems. Finally, we present a real-time implementation of an obstacle avoiding controller for a quadcopter. We demonstrate the results in simulation as well as with real flight experiments.

Micro unmanned aerial vehicles are becoming increasingly interesting for aiding and collaborating with human agents in myriads of applications, but in particular they are useful for monitoring inaccessible or dangerous areas. In order to interact with and monitor humans, these systems need robust and real-time computer vision subsystems that allow to detect and follow persons.In this work, we propose a low-level active vision framework to accomplish these challenging tasks. Based on the LinkQuad platform, we present a system study that implements the detection and tracking of people under fully autonomous flight conditions, keeping the vehicle within a certain distance of a person. The framework integrates state-of-the-art methods from visual detection and tracking, Bayesian filtering, and AI-based control. The results from our experiments clearly suggest that the proposed framework performs real-time detection and tracking of persons in complex scenarios

This chapter presents a distributed architecture for unmanned aircraft systems that provides full integration of both low autonomy and high autonomy. The architecture has been instantiated and used in a rotorbased aerial vehicle, but is not limited to use in particular aircraft systems. Various generic functionalities essential to the integration of both low autonomy and high autonomy in a single system are isolated and described. The architecture has also been extended for use with multi-platform systems. The chapter covers the full spectrum of functionalities required for operation in missions requiring high autonomy. A control kernel is presented with diverse flight modes integrated with a navigation subsystem. Specific interfaces and languages are introduced which provide seamless transition between deliberative and reactive capability and reactive and control capability. Hierarchical Concurrent State Machines are introduced as a real-time mechanism for specifying and executing low-level reactive control. Task Specification Trees are introduced as both a declarative and procedural mechanism for specification of high-level tasks. Task planners and motion planners are described which are tightly integrated into the architecture. Generic middleware capability for specifying data and knowledge flow within the architecture based on a stream abstraction is also described. The use of temporal logic is prevalent and is used both as a specification language and as an integral part of an execution monitoring mechanism. Emphasis is placed on the robust integration and interaction between these diverse functionalities using a principled architectural framework. The architecture has been empirically tested in several complex missions, some of which are described in the chapter.

This paper presents a comparison of two light-weight and low-cost airborne mapping systems. One is based on a lidar technology and the other on a video camera. The airborne lidar system consists of a high-precision global navigation satellite system (GNSS) receiver, a microelectromechanical system (MEMS) inertial measurement unit, a magnetic compass and a low-cost lidar scanner. The vision system is based on a consumer grade video camera. A commercial photogrammetric software package is used to process the acquired images and generate a digital surface model. The two systems are described and compared in terms of hardware requirements and data processing. The systems are also tested and compared with respect to their application on board of an unmanned aerial vehicle (UAV). An evaluation of the accuracy of the two systems is presented. Additionally, the multi echo capability of the lidar sensor is evaluated in a test site covered with dense vegetation. The lidar and the camera systems were mounted and tested on-board an industrial unmanned helicopter with maximum take-off weight of around 100 kilograms. The presented results are based on real flight-test data.

The paper presents a light-weight and low-cost airborne terrain mapping system. The developed Airborne LiDAR Scanner (ALS) sys- tem consists of a high-precision GNSS receiver, an inertial measurement unit and a magnetic compass which are used to complement a LiDAR sensor in order to compute the terrain model. Evaluation of the accuracy of the generated 3D model is presented. Additionally, a comparison is provided between the terrain model generated from the developed ALS system and a model generated using a commer- cial photogrammetric software. Finally, the multi-echo capability of the used LiDAR sensor is evaluated in areas covered with dense vegetation. The ALS system and camera systems were mounted on-board an industrial unmanned helicopter of around 100 kilograms maximum take-off weight. Presented results are based on real flight-test data.

In this paper we consider the problem of searching for an arbitrarily smart and fast evader in a large environment with a team of unmanned aerial vehicles (UAVs) while providing guarantees of detection. Our emphasis is on the fast execution of efficient search strategies that minimize the number of UAVs and the search time. We present the first approach for computing fast search strategies utilizing additional searchers to speed up the execution time and thereby enabling large scale UAV search. In order to scale to very large environments when using UAVs one would either have to overcome the energy limitations of UAVs or pay the cost of utilizing additional UAVs to speed up the search. Our approach is based on coordinating UAVs on sweep lines, covered by the UAV sensors, that move simultaneously through an environment. We present some simulation results that show a significant reduction in execution time when using multiple UAVs and a demonstration of a real system with three ARDrones.

The range of missions performed by Unmanned Aircraft Systems (UAS) has been steadily growing in the past decades thanks to continued development in several disciplines. The goal of increasing the autonomy of UAS's is widening the range of tasks which can be carried out without, or with minimal, external help. This thesis presents methods for increasing specific aspects of autonomy of UAS's operating both in outdoor and indoor environments where cameras are used as the primary sensors.First, a method for fusing color and thermal images for object detection, geolocation and tracking for UAS's operating primarily outdoors is presented. Specifically, a method for building saliency maps where human body locations are marked as points of interest is described. Such maps can be used in emergency situations to increase the situational awareness of first responders or a robotic system itself. Additionally, the same method is applied to the problem of vehicle tracking. A generated stream of geographical locations of tracked vehicles increases situational awareness by allowing for qualitative reasoning about, for example, vehicles overtaking, entering or leaving crossings.Second, two approaches to the UAS indoor localization problem in the absence of GPS-based positioning are presented. Both use cameras as the main sensors and enable autonomous indoor ight and navigation. The first approach takes advantage of cooperation with a ground robot to provide a UAS with its localization information. The second approach uses marker-based visual pose estimation where all computations are done onboard a small-scale aircraft which additionally increases its autonomy by not relying on external computational power.

We present a navigation system for autonomous indoor flight of micro-scale Unmanned Aircraft Systems (UAS) which is based on a method for accurate monocular vision pose estimation. The method makes use of low cost artificial landmarks placed in the environment and allows for fully autonomous flight with all computation done on-board a UAS on COTS hardware. We provide a detailed description of all system components along with an accuracy evaluation and a time profiling result for the pose estimation method. Additionally, we show how the system is integrated with an existing micro-scale UAS and provide results of experimental autonomous flight tests. To our knowledge, this system is one of the first to allow for complete closed-loop control and goal-driven navigation of a micro-scale UAS in an indoor setting without requiring connection to any external entities.

This paper presents a method for high accuracy ground target localization using a Micro Aerial Vehicle (MAV) equipped with a video camera sensor. The proposed method is based on a satellite or aerial image registration technique. The target geo-location is calculated by registering the ground target image taken from an on-board video camera with a geo- referenced satellite image. This method does not require accurate knowledge of the aircraft position and attitude, therefore it is especially suitable for MAV platforms which do not have the capability to carry accurate sensors due to their limited payload weight and power resources. The paper presents results of a ground target geo-location experiment based on an image registration technique. The platform used is a MAV prototype which won the 3rd US-European Micro Aerial Vehicle Competition (MAV07). In the experiment a ground object was localized with an accuracy of 2.3 meters from a ight altitude of 70 meters.

Recent advances in the field of Unmanned Aerial Vehicles (UAVs) make flying robots suitable platforms for carrying sensors and computer systems capable of performing advanced tasks. This paper presents a technique which allows detecting humans at a high frame rate on standard hardware onboard an autonomous UAV in a real-world outdoor environment using thermal and color imagery. Detected human positions are geolocated and a map of points of interest is built. Such a saliency map can, for example, be used to plan medical supply delivery during a disaster relief effort. The technique has been implemented and tested on-board the UAVTech¹ autonomous unmanned helicopter platform as a part of a complete autonomous mission. The results of flight- tests are presented and performance and limitations of the technique are discussed.

Recent advances in the field of micro unmanned aerial vehicles (MAVs) make flying robots of small dimensions suitable platforms for performing advanced indoor missions. In order to achieve autonomous indoor flight a pose estimation technique is necessary. This paper presents a complete system which incorporates a vision-based pose estimation method to allow a MAV to navigate in indoor environments in cooperation with a ground robot. The pose estimation technique uses a lightweight light emitting diode (LED) cube structure as a pattern attached to a MAV. The pattern is observed by a ground robot's camera which provides the flying robot with the estimate of its pose. The system is not confined to a single location and allows for cooperative exploration of unknown environments. It is suitable for performing missions of a search and rescue nature where a MAV extends the range of sensors of the ground robot. The performance of the pose estimation technique and the complete system is presented and experimental flights of a vertical take-off and landing (VTOL) MAV are described.

The use of Unmanned Aerial Vehicles (UAVs) which can operate autonomously in dynamic and complex operational environments is becoming increasingly more common. The UAVTech Lab, is pursuing a long term research endeavour related to the development of future aviation systems which try and push the envelope in terms of using and integrating high-level deliberative or AI functionality with traditional reactive and control components in autonomous UAV systems. In order to carry on such research, one requires challenging mission scenarios which force such integration and development. In this paper, one of these challenging emergency services mission scenarios is presented. It involves search and rescue for injured civilians by UAVs. In leg I of the mission, UAVs scan designated areas and try to identify injured civilians. In leg II of the mission, an attempt is made to deliver medical and other supplies to identified victims. We show how far we have come in implementing and executing such a challenging mission in realistic urban scenarios.

An implemented system for achieving high level situation awareness about traffic situations in an urban area is described. It takes as input sequences of color and thermal images which are used to construct and maintain qualitative object structures and to recognize the traffic behavior of the tracked vehicles in real time. The system is tested both in simulation and on data collected during test flights. To facilitate the signal to symbol transformation and the easy integration of the streams of data from the sensors with the GIS and the chronicle recognition system, DyKnow, a stream-based knowledge processing middleware, is used. It handles the processing of streams, including the temporal aspects of merging and synchronizing streams, and provides suitable abstractions to allow high level reasoning and narrow the sense reasoning gap.

To achieve complex missions an autonomous unmanned aerial vehicle (UAV) operating in dynamic environments must have and maintain situational awareness. This can be achieved by continually gathering information from many sources, selecting the relevant information for current tasks, and deriving models about the environment and the UAV itself. It is often the case models suitable for traditional control, are not sufficient for deliberation. The need for more abstract models creates a sense-reasoning gap. This paper presents DyKnow, a knowledge processing middleware framework, and shows how it supports bridging the gap in a concrete UAV traffic monitoring application. In the example, sequences of color and thermal images are used to construct and maintain qualitative object structures. They model the parts of the environment necessary to recognize traffic behavior of tracked vehicles in real-time. The system has been implemented and tested in simulation and on data collected during flight tests.

This paper describes the approach of the RescueRobots Freiburg team, which is a team of students from the University of Freiburg that originates from the former CS Freiburg team (RoboCupSoccer) and the ResQ Freiburg team (RoboCupRescue Simulation). Furthermore we introduce linkMAV, a micro aerial vehicle platform. Our approach covers RFID-based SLAM and exploration, autonomous detection of relevant 3D structures, visual odometry, and autonomous victim identification. Furthermore, we introduce a custom made 3D Laser Range Finder (LRF) and a novel mechanism for the active distribution of RFID tags.

The use of Unmanned Aerial Vehicles (UAVs) which can operate autonomously in dynamic and complex operational environments is becoming increasingly more common. While the application domains in which they are currently used are still predominantly military in nature, in the future we can expect wide spread usage in thecivil and commercial sectors. In order to insert such vehicles into commercial airspace, it is inherently important that these vehicles can generate collision-free motion plans and also be able to modify such plans during theirexecution in order to deal with contingencies which arise during the course of operation. In this paper, wepresent a fully deployed autonomous unmanned aerial vehicle, based on a Yamaha RMAX helicopter, whichis capable of navigation in urban environments. We describe a motion planning framework which integrates two sample-based motion planning techniques, Probabilistic Roadmaps and Rapidly Exploring Random Treestogether with a path following controller that is used during path execution. Integrating deliberative services, suchas planners, seamlessly with control components in autonomous architectures is currently one of the major open problems in robotics research. We show how the integration between the motion planning framework and thecontrol kernel is done in our system.Additionally, we incorporate a dynamic path reconfigurability scheme. It offers a surprisingly efficient method for dynamic replanning of a motion plan based on unforeseen contingencies which may arise during the execution of a plan. Those contingencies can be inserted via ground operator/UAV interaction to dynamically change UAV flight paths on the fly. The system has been verified through simulation and in actual flight. We present empirical results of the performance of the framework and the path following controller.

While face animation is still considered one of the toughesttasks in computer animation, its potential application range israpidly moving from the classical field of film production intogames, communications, news delivery and commerce. Tosupport such novel applications, it is important to enableproduction and delivery of face animation on a wide range ofplatforms, from high-end animation systems to the web, gameconsoles and mobile phones. Our goal is to offer a frameworkof tools interconnected by standard formats and protocols andcapable of supporting any imaginable application involvingface animation with the desired level of animation quality,automatic production wherever it is possible, and delivery ona wide range of platforms. While this is clearly an ongoingtask, we present the current state of development along withseveral case studies showing that a wide range of applicationsis already enabled.