Cross-Modal Domain Adaptation via Semantic Parametric Maps

 
Transferring perception knowledge across entirely different sensors — from camera image datasets to LiDAR — without overlapping fields of view or any manual labeling, by using a semantic parametric HD map as a bridge. This extends perception from a front-view camera to full 360° coverage while generating training data fully automatically.

Deep learning perception models are only as good as the labeled data they are trained on — and that data is almost always tied to a specific sensor, or even a specific sensor model. As soon as the sensor setup changes or a new modality such as LiDAR is added, existing annotations no longer apply, and manually labeling every new configuration is prohibitively expensive. Established domain adaptation methods only partly solve this: they either require the source and target sensors to be similar in nature, or they rely on an overlapping field of view so that the same object is seen by both sensors at the same moment. Both assumptions break down in realistic vehicle setups.

Towards this goal, we present XD-MAP, a self-supervised approach that transfers knowledge across fundamentally different sensing modalities using a semantic HD map as a bridge. A well-generalizing neural network detects static landmarks — poles, traffic lights, and traffic signs — in ordinary camera images. Using a highly accurate feature-based SLAM, these detections are accumulated over multiple drives into a compact, semantically tailored map, in which each class is represented by the geometric primitive that fits it best: cylinders for poles and traffic lights, and shaped planes for signs. Because this map lives in 3D and is decoupled from any single sensor, its elements can then be rendered into a completely different target sensor — in our case a 360° LiDAR — producing dense, pixel- and point-accurate pseudo labels without a single manual annotation.

This map-as-a-bridge design removes the two central limitations of prior work. Since no overlapping field of view is required, knowledge captured by a front-view camera (covering roughly 100°) can be transferred to the full 360° coverage of a surround LiDAR — effectively teaching a sensor to recognize objects in directions the source camera never observed. The same map supplies ground truth for several tasks at once, including 2D semantic segmentation, 2D panoptic segmentation, and 3D semantic segmentation, and it can just as easily be projected back into other cameras. In effect, this makes the rich knowledge embedded in large, well-generalizing image datasets available to underrepresented or specialized sensors such as LiDAR, turning existing map data into a scalable, reusable source of training data. Across all three tasks, models trained on these pseudo labels clearly outperform single-shot baselines constrained by the camera's field of view.

Building on our Learning from Maps project, this line of work is a step towards fully automated, sensor-agnostic training-data generation. Ongoing work extends it to further semantic classes, to dynamic objects that require 4D reconstruction, and to additional sensors such as imaging radar.
 

Publications

Frank Bieder, Hendrik Königshof, Haohao Hu, Fabian Immel, Yinzhe Shen, Jan-Hendrik Pauls, Christoph Stiller. XD-MAP: Cross-Modal Domain Adaptation via Semantic Parametric Maps for Scalable Training Data Generation. IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Denver, CO, USA, 2026.

Frank Bieder. Learning from Maps: Scalable Ground Truth Generation in Autonomous Driving. PhD thesis, Karlsruher Institut für Technologie (KIT), 2026. doi: 10.5445/IR/1000192685.