This short article explains how to submit your mono depth method to the rabbitAI depth prediction benchmark.
Read at the bottom of the page for regular (non-docker) submission! If you have question contact us via email at benchmark@rabbitai.de.

Current Submission Rules

1. Only one submission per method. Additional submissions will be granted in case of obvious technical errors (e.g. mixed up filenames). Additional submission may also be possible for significant variants of individual methods (e.g. pure supervised vs unsupervised), though may be hidden/removed over time to unclutter the list.
2. Methods must be associated with some publications, either a peer-reviewed paper or publicly available source code and description. If your work is in review or just being submitted write down where and when the reviews are expected.

General Remarks

There are two possible ways to submit your mono depth results to the rabbitAI benchmark. You can either submit your method as an executable Docker image or download the rabbiAI benchmark data and submit your results directly. In both ways, you should submit a single compressed folder here. Both possibilities are explained in more detail in the following sections.

Input and Output Format

The rabbitAI training and evaluation images are provided as 8 bit RGB PNG images with a resolution of 1216 x 352, which closely match the images of the KITTI Depth Prediction Benchmark.

The output depth predictions should be provided as 16 bit greyscale PNG images with the same resolution and with pixel values representing the depth in meter multiplied by 256, i.e. a pixel value of 256 corresponds to a predicted depth of 1 meter.

Submitting a Docker image

Docker Installation

For installing Docker on your machine please follow the instructions on the official Docker documentation.

How your docker image should look like

For the rabbitAi benchmark you can submit a Docker image that can be used to build a running Docker container. To allow an easy integration into the existing benchmark, your image should meet the following standards.

Your Docker image should contain a /images directory that we can use to mount the benchmark images into your container. Moreover, it should contain a /results directory that we also mount externally on runtime and which will store the results of your mono depth method. Both directories should be empty initially and will be mounted using the command:

 docker run -v <external image folder>:/images -v <external results folder>:/results --rm <image>

In this way we are able to pipe our benchmark data into your Docker container and retrieve the resulting images. Your Docker image should contain an automatic entrypoint such that it can be used as an executable.

The entrypoint should run the mono depth prediction on all images in the container’s /images directory and store the results in the /results directory preserving the filenames of the /images folder. The latter point is of great importance as it allows us to easily match the input images and the resulting depth predictions. These will be used to evaluate your results with the metrics of the rabbitAI benchmark.

It is thus only necessary for you to supply executable evaluation code.

To submit your docker image request an upload link here.

Submitting Mono Depth Predictions without Docker

If you do not want to submit your method as an executable Docker image, you alternatively can submit your mono depth predictions directly.

Download the rabbitAI benchmark submission images and submit your mono depth predictions as a zip file by requesting an upload link here. The submitted depth images should preserve the filenames of the input images.

If you have question contact us via email at benchmark@rabbitai.de.