Submission Instructions
Prepare JSON Submission File
The downloaded data only contains annotations on train and validation sets. The annotations of the test set are held out. To obtain the performance on the test set of StereOBJ-1M, the user needs to submit to StereOBJ-1M Challenge on EvalAI.
Users need to submit their pose estimation results on the StereOBJ-1M test set for 17 of the 18 objects (all objects except centrifuge_tube). To submit to StereOBJ-1M Challenge, please prepare a submission JSON file that contains the 6D pose prediction results. The required format of the JSON is as follows:
{
"split": "test",
"pred": {
"blade_razor": {
"mechanics_scene_1_07272020_13": {
"000000": [
[1., 0., 0., 0.],
[0., 1., 0., 0.],
[0., 0., 1., 0.],
],
"000001": [
[1., 0., 0., 0.],
[0., 1., 0., 0.],
[0., 0., 1., 0.],
],
...
},
"mechanics_scene_2_08012020_17": {
"000000": [ ...
],
...
},
...
},
"hammer": {
"mechanics_scene_11_08212020_3": {
...
},
...
},
},
}
The value of the split key should be the dataset split (i.e. "test").
The value of the pred key should be each object's 6D pose predictions in each frame of each sequence, indexed by object class name (e.g. "blade_razor"), sequence name (e.g. "mechanics_scene_1_07272020_13"), and frame index (e.g. "000000") from top to bottom in the hierarchy. The 6D pose should be represented by a 3x4 matrix of [R|t], where R is the 3x3 rotation matrix and t is the 3x1 translation vector.
We provide an example in our baseline code on how to generate the submission file. Please refer to our code for more details. We recommend that the users first run the evaluation locally on the validation set using our evaluation script to make sure that the JSON file format is correct before submitting it to the challenge.
Submit to StereOBJ-1M Challenge EvalAI
Please register an account on EvalAI and submit your results in JSON file to our challenge. You need to create a team and submit the prepared JSON file. It takes about 45 minutes for the server to evaluate the submitted JSON file and return the results, so please plan early if you are working towards a deadline.