The PENGWIN Challenge consists of two independent tasks.
Participant are allowed to choose one of them or both. Each task
has a Preliminary Development Phase and a Final Test
Phase. Training sets are now accessible on Zenodo for all
participants.
The test data will not be opened to the participants. For final
submissions, participants must use a Docker container. A
template for the Docker container will be made available prior to
the opening of submissions.
During the Preliminary Development Phase, participants may make up
to five submissions to perform sanity checks using a small
validation set. In the Final Test Phase, only one submission per
participant is permitted.
Usage of publicly available data including (open) pre-trained
nets is allowed.
Participants are required to use their real names and affiliations
on their Grand Challenge profiles. Entries from individuals with
multiple or duplicate profiles will result in disqualification.
Participants may develop their methods using pre-trained AI models
or any publicly available datasets, provided these are under a
permissive license. The use of such resources must be explicitly
stated in their submissions.
Top three performing teams are required to share their code on a
public repository.
Participants are allowed to publish their results separately after
the challenge. The publication embargo period will be 12 months
after the announcement of the challenge results.
The primary metrics for evaluation are Intersection over Union (IoU),
Hausdorff Distance at 95% (HD95), and Average Symmetric Surface
Distance (ASSD).
The predicted fragment instances and the ground-truth labels are
matched based on the highest IoU.
Evaluation is perfromed on both fracture segmentation result
(comparing matched fragment instance and ground truth) and anatomical
segmentation result (labels for each bone are merged and compared to
the ground-truth anatomical labels of sacrum, left hipbone, and right
hipbone). Therefore, there are six final metrics: fracture_iou,
fracture_hd95, fracture_assd, anatomical_iou, anatomical_hd95, and
anatomical_assd.
The metrics are ranked indepentedly. The final rank for each
submission is determined by averaging the ranks on single metrics.
