Move submission log storage from temp dir to Redis
Overview
To improve end user experience and not drop logs between deployments, we should store logs in Redis.
Relevant sections of code:
This websocket endpoint takes a list of submission IDs and returns their data from temp file storage. This needs to be changed to use redis instead of temp file storage.
async def receive(self, text_data=None, bytes_data=None):
"""We expect to receive a message at this endpoint containing the ID(s) of submissions to get
details about; typically on page load, looking up the previous submission details"""
data = json.loads(text_data)
submission_ids = data.get("submission_ids", [])
if submission_ids:
# Filter out submissions not by this user
submissions = Submission.objects.filter(id__in=submission_ids, owner=self.scope["user"])
for sub in submissions:
text_path = os.path.join(settings.TEMP_SUBMISSION_STORAGE, f"{sub.id}.txt")
if os.path.exists(text_path):
with open(text_path) as f:
text = f.read()
await self.group_send(text, sub.id, full_text=True)
Tasks
-
Make websocket endpoint read from redis -
Make websocket endpoint write to redis -
Ensure redis on Heroku is configured to dump the oldest data when we go over data limits