3D Reconstruction meets Semantics 2018 – Challenge


In order to support work on these questions, the workshop features a semantic reconstruction challenge. The dataset was rendered from a drive through a semantically-rich virtual garden scene with many fine structures. Virtual models of the environment will allow us to provide exact ground truth for the 3D structure and semantics of the garden and rendered images from virtual multi-camera rig, enabling the use of both stereo and motion stereo information. The challenge participants will submit their result for benchmarking in one or more categories: the quality of the 3D reconstructions, the quality of semantic segmentation, and the quality of semantically annotated 3D models. Additionally, a dataset captured in the real garden from moving robot is available for validation.


Given a set of images and their known camera poses, the goal of the challenge is to create a semantically annotated 3D model of the scene. To this end, it will be necessary to compute depth maps for the images and then fuse them together (potentially while incorporating information from the semantics) into a single 3D model.

We provide the following data for the challenge:

  • A synthtetic training sequences consisting of
    • calibrated images with their camera poses,
    • ground truth semantic annotations for a subset of these images,
    • a semantically annotated 3D point cloud depicting the area of the training sequence.
  • A synthetic testing sequence consisting of calibrated images with their camera poses.
  • A real-world testing sequence consisting of calibrated images with their camera poses.

Both training and testing data are available here. Please see the git repository for details on the file formats.

This year we accept submissions in several categories: semantics and geometry, either joint or separate. For example, if you have a pipeline that first computes semantics and geometry independently and then fuses them, we can compare how the fused result improved accuracy.

A. Semantic Mesh

In order to submit to the main category if the challenge, please create a semantically annotated 3D triangle mesh from the test sequence. The mesh should be stored in the PLY text format. The file should store for each triangle a color corresponding to the triangle’s semantic class (see the calibrations/colors.yaml file for the mapping between semantic classes and colors).

We will evaluate the quality of the 3D meshes based on the completeness of the reconstruction, i.e., how much of the ground truth is covered, the accuracy of the reconstruction, i.e., how accurately the 3D mesh models the scene, and the semantic quality of the mesh, i.e., how close the semantics of the mesh are to the ground truth.

B. Geometric Mesh

Same as above, but PLY mesh without semantic annotations.

C. Semantic Image Annotations

Create a set of semantic image annotations for all views in the test, using the same filename convention and PNG format as in the training part. Upload them in a single ZIP archive.


The deadline for submitting to the challenge is August 31th (23:59 GMT).

Once you have created the output, please submit it using this link, one file per category and dataset. Please use unique filenames to indentify yourself and result type, eg. smith_method_mesh_B_synthetic.ply.

In addition, please send a summary email to rtylecek@inf.ed.ac.uk that includes

  • the filenames of the file you submitted,
  • synthetic or real dataset,
  • challenge category (A/B/C),
  • the label for your entry, eg. method or group name.

For questions, please contact rtylecek@inf.ed.ac.uk.

RT @FICbv: Meer samenwerking tussen parallelle researchprojecten is nodig voor versnelling, aldus Angelo Mencarelli van @trimbot2020 op @To…2 weeks
@trimbot2020 mentioned in the magazine #Mikroniek of precision @engineering1!! Check out all the mentions at https://t.co/rsUKtcCkeb2 weeks
RT @DAgriFood: 🛰🚜🌾 EU’s future cyber-farms to utilise #drones, #robots and sensors ▶️ https://t.co/99rzcR0D4H @HorizonMagEU @IFParis @IoF20…5 months
#Pictures from the last consortium meeting in Renningen #robotics #trimbot #H2020 @EU_H2020 @RoboticsEU https://t.co/6uRcXqth7M6 months
TrimBot2020's robotic arm cuts bushes #robotics #automatic #cutting #bush #gardening #research #sterevision… https://t.co/SoWhNc4SXm6 months