Depth Prediction Benchmark

overview

Welcome to the rabbitAI depth prediction benchmark. Please see our paper (cite) for a detailed description.

Training data is available, submission instructions can be found here, register for updates. \

This benchmark will be part of the Robust Vision Challenge (2020).

robust_vision_banner

News

  • 2020-07-06 We have added another pretrained reference method: MiDaS
  • 2020-06-14 Submissions are now open: submit your algorithm or result
  • 2020-04-08 Training data is now available
  • 2020-04-06 Our paper has been accepted to SAIAD 2020 (preprint/bibtex)
  • 2020-04-03 We fixed a bug in the accounting of all metrics, results have changed slightly, bump30 changed significantly
  • 2020-03-22 Public leaderboard is online

Leaderboard

Public leaderboard, sorted by Avg30. Click on the method name to see detailed results for the method.

Network avg30 miss30 fake30 missSt30 fakeSt30 bump30 Avg ScaleError Avg Offset [m] silog [%] sq_rel [%] abs_rel [%] rmse_inv [1/km]
SemiDepth 0.26 0.37 0.45 0.20 0.08 0.18 0.96 2.58 38.58 26.35 26.56 38.54
DenseDepth 0.32 0.53 0.41 0.25 0.04 0.35 1.20 3.88 38.26 13.5 24.94 35.5
Midas 0.42 0.78 0.33 0.27 0.04 0.69 1.12 16.33 48.05 20.51 32.92 43.83
MonoResMatch 0.46 0.27 0.82 0.24 0.43 0.55 1.03 3.25 43.33 21.78 29.2 48.49
BTS 0.51 0.25 1.00 0.22 0.92 0.14 0.94 2.42 51.09 15.91 27.38 50.41

Distance Metrics

The following plots evalute our interpretable metrics over a range from 3 to 100 meters. See the the paper for details. Note that all metrics only operate in the drivable corridor up to 2m above the street surface.

Miss

The Miss metric detects obstacles (on and off-street) that are missing in the algorithm result. For example parked cars, trees, barriers, bollards, buildings. miss plot

Fake

The fake metric detects obstacles (on and off-street) that are hallucinated in places which should be empty (and hence drivable). fake plot

MissSt

Same as Miss, but restricted to obstacles directly above the visible street surface (e.g. boom gates, branches, side mirrors). missSt plot

FakeSt

Same as fake, but restricted to the area directly above the visible street surface. fakeSt plot