Transparency of docker images and scoring programs

It is shown in #723 (closed) that the starting kit and public data buttons do not exist anymore in Codabench.

More generally, we would like to add a "transparency" feature. Organizers could have option to decide what they want to make public: the competition docker image, the scoring program, ingestion program, input data, etc.

Each piece made public would have a download button from the competition web pages. The docker image could simply be shown using its DockerHub tag.

The sample submission (solution) should also be subject to this transparency, not only by being downloadable, but by being automatically run and evaluated.

The fact that CodaLab Competitions and Codabench are open-source is a crucial point in making these platforms great tools for the scientific community. This way, everybody can review the code and the benchmarks and competitions are more valuable. However, the code and data used by the organizers is not necessarily open. Even if some organizers provide the code via Github with a link in the documentation, one can not ensure that it is really the code used to evaluate the submissions, making the final results less transparent. The goal of this feature is to have a real transparency, when enabled, between the organizers and the participants.

---

Similar ideas:

• Organizers contact email

It would be good to have a place, typically a field in the editor, for the organizer's email address. This is important because it is very common that participants contact the administrators of the platform instead of the organizers for competition specific questions because they can't find their contact.

Inside parenthesis next to "organized by":

A sentence in the forum:

• Link to the report

It would be interesting to have a way to show the link to a competition report. The link could be accessible from an icon benchmark listing, and also well visible in the competition home page.

• Reward (prize) of the competition

As stated in #954, we should be able to set a reward (e.g. 4000$) from the editor. I don't think we want it to be doable from the competition.yaml file because that should be done only one the "production" version of the competition, after the tests. It could then be shown in the interface, in the home page of the challenge and in the public challenges list.

In the list:

Below "participants" and "submissions" using the same styling:

TODOs:

Editor / YAML fields

• Display docker image name in competition Solved by: #944
• Add a YAML and editor field Organizers contact where the organizers put an email address that will be shown in the interface of the competition Solved by: #1063
• Editor field Reward (free-form string?) Solved by: #1063
• Editor field Challenge report Solved by: #1063

Files tab

First PR #970:

• For non-approved participants: Not accessible
• For approved participants: Showing objects made available by organizers
• For organizers: Showing all object and a column showing what is available or not to participants

Next TODOs:

• In editor / YAML, two options: make programs available, make input data available (#1136)
• Add documentation about the fields to add starting kit and public data in the file competition.yaml
• Solutions appear even when not registered to the competition. Should be treated in the same way as public data and starting kit (show only to registered participants and organizers, and always in available state)
• The No Files Available Yet text is always showing (#1041)
• Fix the column Available that always displays "Yes" (see #970) (#1136)
• Green checks instead of yes (see screenshot below, like the resources interface). These ticks are just markers and should not be clickable. (#1136)

See the discussion below for better understanding