[Feature] Meta scoring program: organizer provides the code for computation between leaderboard columns
Competition organizers can define their own scoring and ingestion programs in a flexible way with Python code.
But, in Codalab v15, the computation of the final ranking when there are several sub-scores is constrained between few possibilities (average rank, maximum...).
I think it is crucial to be able to use any function to compute this final ranking, so organizers should be able to provide a kind of "meta scoring program".
The same idea was mentioned here:
- #445
Much more flexible way to handle computation columns. This should be a fine solution for a much more advanced "computational column" type system. Organizers could do any kind of computation they'd like! This should be executed even if children have failed.
Also, a similar feature we'd like:
- #444
Another related issue:
- #447
Currently, there is one column(primary column) that is used to decide ranking on the leaderboard. We should add ways to compute leaderboard ranking across all columns on the leaderboard, and have a way to exclude to certain columns from the ranking calculation. Make computed columns.
- Could automatically be generated by columns with the same key - Pro: simple and programmatic - Cons: Would have less control
- Could require each column to have a unique key and then group the columns later to be calculated OR - Could instead select columns by key + task - Pro: More control in selection - Con: tasks more work by admins
And the following post-it / discussion issue:
- #443