Institut für Mess- und Regelungstechnik (MRT)

Localization and Mapping

Group Leader: M.Sc. Jan-Hendrik Pauls

Automated driving still depends on highly accurate maps that provide enriched layered knowledge on e.g. lane geometries, traffic rules and traffic lights. This information is not always perceivable or the surroundings do not allow safe detection - may it be due to bad weather or due to occlusions. The research group Mapping & Localization investigates challenges around map-based automated driving and tackles the future vision of mapless driving.

When using highly accurate maps, multiple challenges occur. At first, the vehicle's ego-motion needs to be determined (Visual/Lidar/Inertial Odometry) to properly reconstruct the world around. We create rich and dense 3D representations as well as sparse abstract or semantic feature maps that are compact in storage and lean in maintenance. These maps do not only contain physical elements, but also higher order information, such as lanes, routes or traffic rules. The process to derive a suitable representation from sensor data is called Mapping.

To use a map for automated driving, the automated vehicle has to be localized in the map, so static map data can be associated with detected dynamic information and used for driving (Localization).

While maps represent the static world, seemingly static parts of the world in reality change quite frequently. These changes have to be detected (Map Verification) so the automated vehicle is able to drive safely only relying on correct information.

In all cases, where a map is outdated or the localization is misleading, a Mapless Driving solution has to be established as fallback solution. We aim at infering map data with a variety of strategies with e.g. deep learning based semantic segmentation and model-based lane inference.

Finally, highly accurate localization in a verified map allows to use map information to generate infinite ground truth without extra labeling effort. This enables the field of Map Learning to fully automate the process of mapping.



Learning from maps : Automatic data generation for deep learning applications using HD maps and multi-drive mapping

Our highly accurate localization in a verified map allows to use map information to generate infinite ground truth without extra labeling effort. We leverage our multi-drive mapping algorithms to record as much sensor data as needed to generate a dataset with a suffienct size and variety in weather conditions to fullfil the task at hand. The data is generated by back-projecting 3D objects into the sensor measurements.

Contact: M.Sc. Frank Bieder


Life-Long Multimodal Continuous Mapping and Localization

For automated vehicles, high definition maps (HD maps) are required to overcome shortcomings of the online perception system. For the HD maps generation, the estimation of highly accurate ego-vehicle motion is the key technique. Common approaches, which only use inertial measurement units (IMU) with global navigation satellite systems (GNSS) can not provide accurate ego-motion especially in urban areas due to shadowing and non-line-of-sight (NLOS) effects.

To reach this goal, we investigate in the Life-Long Continuous Mapping and Localization using IMU, cameras and LiDAR mounted on the vehicle. The motion data from different sensors is fused in a time continuous way. The output of our system is a high accurate continuous trajectory with local and global consistency. Using this trajectory, several applications like 3d reconstruction, HD maps generation and automated training data generation can be applied.

Contact: M.Sc. Haohao Hu


Meaningful features for localization and more

A common approach for localization is to use abstract features detected in e.g. camera images or Lidar scans. These features enable precise localization but have no other use. We develop meaningful features for localization which can further be used for other tasks such as planning or behavior generation.

Groundtruth path (red), localization result (orange arrows), detections of facades, poles and road markings.

Contact: Dr. rer. nat. Martin Lauer


Direct Localization in HD Planning Maps

To directly localize in a high definition (HD) planning map, using only a monocular camera, we utilize the results of recent neural networks. Apply distance transform on binary images for each detected class and reprojecting corresponding map elements using an initial pose guess allows to optimize the vehicle pose by minimizing the resulting error term. The distance images serve as fast look-up tables during optimization. Combining multiple frames with vehicle odometry in a sliding window pose graph allows to overcome single frame classification errors and results in state-of-the-art localization performance.

Contact: M.Sc. Jan-Hendrik Pauls


Automatic Verification of High Precision Maps for Highly Automated Driving

The recent progress in the area of autonomous cars has shown that high precision digital maps are crucial to steer a car safely and comfortably through a complex dynamic environment. To plan a trajectory which guides a self-driving car as smoothly as a foresightedly driving human driver would do, details on a sub-lane level are needed. However, the more details are stored in a map, the faster it becomes outdated.
Thus, the goal of our research is to use sensor data from sensors, which are needed for autonomous driving, anyway, to locally verify the map that is stored on the car. This can either verify the map or mark it - or parts of it - as invalid.
As part of our work, we are developing methods and a framework to compare map and sensor data. Furthermore, we are trying to identify features that are suitable for map verification. Another challenge is to model static and dynamic occlusions which limit the range within which the map can be assessed.
Parts of the map which are eventually marked as changed can not only be invalidated for all other components of an autonomous car, but also sent back to a remote server. When permanent changes are identified, they could either trigger a remapping process or be used as a map update directly.

Contact: M.Sc. Jan-Hendrik Pauls


Realtime Intersection Estimation for Mapless Driving and Map Updates

Modern automated vehicles heavily rely on highly accurate maps. Maps are prone to become outdated due to constructions and changes in the lane system. These cases will result in unexpected behavior, if the vehicle is not able to navigate without a precise map. In addition, updating existing maps is not even covered in recent research, yet. Thus, our research goal is to estimate map data in complex, urban environments solely based on sensor information. The resulting estimation can provide missing planning constraints and simultaneously update outdated maps. For providing reliable intersection models, we combine Deep Learning based methods with model based sampling and optimization techniques.

Contact: M.Sc. Annika Meyer


Automatic Generation of High Precision Maps for Automated Driving

For the operation of safe and reliable automatic vehicles, high-resolution maps, which also contain detailed information about, for example, lanes and their exact position, information about right of way and speeds limits, are indispensable. To reduce the high effort associated with the creation and validation of such maps, we are developing methods to automate this process based on sensor data from measurement vehicles.

Contact: M.Sc. Jan-Hendrik Pauls


Large-scale 3D scene reconstruction and texturing

3D reconstructions can be used for localization, simulation, visualization and many more tasks. Modern LiDAR sensors provide a large amount of sub-centimeter accurate point measurements of their environment. We fuse this data using a volumetric reconstruction approach that allows for the reconstruction of large areas. The challenge is to incorporate data from multiple drives in a way that improves the reconstruction. In order to do so, loop closures have to be detected and all sensor poses have to be determined in a way that results in a consistent model of the environment. After the geometry has been reconstructed, it can be textured from camera images. The following video shows results using data from the KITTI dataset.

Contact: M.Sc. Tilman Kühner


Lane marking based localization on Highways

Lane-level accurate localization is essential for autonomous driving on Highways. Using low-cost mono cameras, we detect lane markings in the current camera images with the map to obain the position of the vehicle.

Contact: M.Sc. Johannes Janosovits


Life-Long Vision based Mapping and Localization

Current intelligent vehicles require robust and accurate self-localization in a multitude of scenarios. Common approaches couple inertial measurement units (IMU) with global navigation satellite systems (GNSS). However, such solutions are not reliable in urban environments due to multipath, shadowing and atmospheric perturbations.

To overcome these drawbacks we investigate in life-long iterative mapping and high-precision localization in six degrees of freedom using multiple cameras mounted on the vehicle. The approach yields centimeter accuracy even under challenging conditions.

Contact: M.Sc. Haohao Hu