Add caching for datasets downloaded in the compute worker
how we did this in v1.5: https://github.com/codalab/competitions-v1-compute-worker/blob/162-nvidia-worker/worker.py#L131
Hash the URL Check if hash exists in cache folder If not: Download + save the dataset to that hash filename else: return reference to existing file
Clean up automatically after N gig limit reached