Segment Me If You Can

Road Anomaly Benchmark

Meet the Team!

Robin Chan

Krzysztof Lis

Svenja Uhlemeyer

Hermann Blum

Sina Honari

Matthias Rottmann

Deep CNNs are unreliable outside of their training distribution
Perception Failures were at the Heart of Past Accidents
Data for Anomaly Segmentation is Scarce
Public Leaderboard
Anomalies can appear everywhere in the image.
Anomalies widely differ in size.
Wide variety of environments.
Labeling Policy

Decision whether an object is anomalous or not is based on the 19 Cityscapes evaluation classes. If an object can be assigned to one of these classes, it is labeled as not anomaly (white), otherwise as anomaly (orange) or void (black).

The objects of interest are the carriage including the horses. The other unknown objects in the background (e.g. parasols, chair) are voided. We also void small gaps inside of the anomalies.
Obstacles appear at different distances.
Different road surfaces.
Different lighting and weather conditions..
Labeling Policy

The road ahead is the region of interest, labeled as not obstacle (white). Every object placed on the road is labeled as obstacle (orange), everything besides the road is voided (black). We also void areas that could distract the model, such as wet spots on the road.

The road is the region of interest. The boot is labeled as obstacle, the background and the wet spots are ignored in the evaluation.
Anomaly Segmentation Performance Metrics
  • Classic pixel-wise metrics:
    • AUROC: Area under receiver operating characteristic curve (TPR vs. FPR)
    • AUPRC: Area under precision recall curve (precision vs. recall)
  • Recent component-wise metrics:
    • sIoU pred: adjusted component-wise intersection over union wrt pred
    • sIoU gt: adjusted component-wise intersection over union wrt ground truth
    • TP: sIoU gt greater than a given threshold
    • FN: sIoU gt smaller than a given threshold
    • FP: sIoU pred smaller than a given threshold
    • F1 := 2TP / (2TP + FP + FN)
ordinary vs. adjusted component-wise intersection over union (IoU vs. sIoU)