The Wallenberg laboratory for research on
Information Technology and Autonomous Systems

Demonstrated System - December 1998

The WITAS project started on January 1, 1997, and is planned for a first phase of three years (1997-1999) and a second phase of four years (2000-2003). The project has two goals with equal dignity:

Construct a particular kind of system -- the onboard IT system whereby an unmanned aerial vehicle can "think" for itself -- for demonstration by the end of the year 2003.
Conduct disciplinary research on topics that are of importance in the international research community, and that are relevant for the system being built. This research involves both theory and methodology.

It is this combination of building a complex system and doing disciplinary research that gives the WITAS project its special character. The present memorandum summarizes the results towards the systems goal during WITAS Phase I. Additional details and descriptions of other aspects of the project are found at the WITAS website,

    http://www.ida.liu.se/ext/witas/

Background and general directions for the systems development

The WITAS developed system must then consist of the following essential components or subsystems:

On-board computer system
Computer vision system (video and IR cameras, camera housing for stabilization, etc)
Other on-board sensors
Communication with an operator on the ground, or in another vehicle.

Such a flying vehicle is usually called a UAV, for Unmanned Aerial Vehicle. The UAV in our project is intended for a particular operational environment, namely, for roads carrying automobile traffic. It is therefore required to "understand" what happens on those roads - conventional maneuvers of individual cars and other road vehicles, dangerous or otherwise exceptional maneuvers, structure of the traffic e.g. congestion. It is also to perform tasks that are assigned by the operator or triggered by the observations it makes itself, for example to follow a certain car that flees from the scene of an apparent crime, or to assist a certain car so that it can make it through difficult traffic and get to a particular destination as quickly as possible, or to deliver a particular parcel to a particular point.

The UAV is supposed to perform these functions autonomously, that is, without the direct intervention of a human operator. It is therefore not sufficient to design it for remote control of its maneuvers and other detailed operations: the operator is only supposed to communicate general commands, often using a combination of a phrase in natural language, and pointing to a map or video image. The most important capabilities for such a system are therefore (1) to form a model of ("understand") scenes and events that it observes on the ground, and (2) prediction, planning, and autonomous decisions using that model.

System Behavior Goals

For a project of this kind it is important to have a clear definition of the system one intends to build. The system can be defined in various ways (physical characteristics, what are the major subsystems, and so on), but the most important is the specification of the behavior that the developed system is required to exhibit and the scenarios that it is required to master. These definitions were made already in the original project proposal from 1996, where they were stated as follows:

Fly over a congested highway system (typically in a city) and detect traffic accidents by the pattern of how cars are standing and/or moving
Fly over a congested highway system and find the best route for an emergency vehicle (avoiding traffic jams)
Fly over roads in a segment of countryside where there have been hostilities (but fighting has stopped), identify how roads have been blocked (by obstructing vehicles, destroyed bridges, etc), and identify a route where trucks can pass
``Watch that car'' -- fly over a particular car (for example a high-value transport) and supervise what happens around it. If there seems to be any trouble, then fly down and take close-up photographs; also report back to a communication center.
``Follow that car'' -- if there has been an incident in the activity of the previous item, then follow a particular fleeing car and check where it goes. If the people in it switch to yet another car, then watch what happens and follow the (a) car that moves rather than (the) one that stays.
Interception analysis: In the context of the car-watch and car-following activities, do lookahead across the road network to predict if the watched car or the fleeing car will be forced to stop because of an obstacle. Also, identify ways of stopping it using available, cooperating vechicles (police cars, etc).

The original project proposal continued as follows:

"The proposal, then, is to develop an IT system (sensors, computer hardware and software, radio communication) where all of these functions are performed automatically: the on-board vision sensors observe the roads and situations under the aeroplane and analyze the images, and the decision system specifies where to fly and which communication to have with the base. (...)"

Essential aspects of the end-1998 demo

Since the project started on January 1, 1997, we have had development milestones in December 1997, May 1998, and December 1998. For each milestone we produced a video showing the accomplishments of the system that had been developed so far.

During calendar year 1999 we put a larger emphasis on theoretical and other foundational work. With the start of Phase II in year 2000 we have come back to system implementation issues, including the building of the on-board system for actual (non-simulated) operation. This is however outside the scope of the present memo.

The following are the essential aspects of the demo completed in December, 1998:

A simulated UAV flies in a simulated operational environment ("virtual reality") of the kind that is defined in the WITAS challenge, that is, over a system of roads where cars move around.
The UAV simulation is based on a quite precise, mathematical model of a helicopter, which has been obtained from contemporary literature and under the direction of an expert from Saab Dynamics, civ.ing. Erik Skarman.
The control system of the simulated UAV contains the required levels of control, up to a "deliberative" layer where the system is able to predict the possible actions of cars that it observes, and to take "intelligent" action with respect to them.
The simulated UAV also contains a computer vision system that analyzes images that are obtained from the virtual reality. The vision system provides the results of that analysis to the cognitive layers of the architecture. It makes use of new advances in computer vision technology. Parts of it have been structured as a set of relatively independent "skills", thus moving away from the traditional concept of a monolithic vision system. In this way the higher levels of the WISS architecture are able, more than before, to control the allocation of computational resources within the scene understanding process.
The computer vision system has been developed on the basis of research frontline techniques that must be run in laboratory environments and on very powerful computers when they are in their development stage. In our system, some parts of the computer vision algorithms have been transferred to a compact computer system that weighs only a few kilograms, and that can therefore realistically be brought on board a small conventional UAV. As the project proceeds, all the vision algorithms will be transferred to the on-board computers. Further minimization using various VLSI techniques is also foreseen.
The UAV simulation and the virtual operational environment are constructed using a combination of commerically available software tools and the results of our own research. The grahical rendering software is a standard, commercial product. The project develops a modelling system and language, Modelica, for describing complex, object-oriented systems. In this respect it particpates in an international, cooperative effort that is intended to lead to standardization.
The developed system has a clean, well defined, and well documented architecture that lends itself to the replacement of modules and the extension with additional subsystems. In particular, modules communicate by TCP/IP connections, which means that different modules can be located on different nodes in the Internet, even at arbitrary distance from each other. The architecture therefore provides an excellent platform for the continued reseach.

Relationship to the disciplinary research part of WITAS

The other part of WITAS, besides systems building, is the disciplinary research, whose primary means of reporting is through published journal and conference articles. A list of these publications from WITAS Phase I is published separately and is available on the WITAS webpage.

Some of the publications are primarily descriptions of some aspects of the WITAS systems development. This applies e.g. for the following articles (with the numbers used in the publication list): 1998:026, 1998:163, and 1999:051.

However, most of the publications describe disciplinary research results that have not directly contributed to the system that was built during 1997 and 1998. In many cases the results will be used in the next generation system being built in phase II. It is interesting, however, that there is often a connection in the opposite direction, where the experience from the systems building in phase I is part of the background for the research publications during those years. Schematically, we can see a chain of the following form:

     Systems implementation during phase I (1997 - 1998)
        leads to
     Generalization of principles published during phase I (1998 - 1999)
        leads to
     Application of these principles for systems implementation
     during phase II (2000 and onwards)

We believe that this is a natural and sound way of connecting systems work and disciplinary research.

In addition, of course, the publications during phase I contain a number of more theoretical results that are expected to go into the phase II system.

Additional documentation

An exhaustive description of the WITAS system architecture from the project's phase I is fond in the following reference:

Patrick Doherty. The WITAS Integrated Software System Architecture. Published by Linköping University Electronic Press and available at http://www.ep.liu.se/ea/cis/1999/017/.

Maintenance information:
Latest update 6.11.2000 by WITAS secretariat.
Edit mode aml, position code D.witas.info.demos.
To proceed, click other item in menue