Skip to content

Storage Analytics scan failing + wrong dates

Hello, I have 2 problems related to Storage Analytics to report.

Problem 1:

  • Solved #1465

There seems to be a problem with the Storage Analytics function.

When the function is run, it fails immediately with the following error :

[2024-05-15 08:50:42,369: INFO/MainProcess] Received task: analytics.tasks.create_storage_analytics_snapshot[fb0db7bd-5e81-4e1c-9946-723da5f860ef]  
[2024-05-15 08:50:42,372: INFO/ForkPoolWorker-2] Task create_storage_analytics_snapshot started
[2024-05-15 08:50:44,205: ERROR/ForkPoolWorker-2] Task analytics.tasks.create_storage_analytics_snapshot[fb0db7bd-5e81-4e1c-9946-723da5f860ef] raised unexpected: ClientError('An error occurred (404) when calling the HeadObject operation: Not Found')
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 382, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 641, in __protected_call__
    return self.run(*args, **kwargs)
  File "/app/src/apps/analytics/tasks.py", line 49, in create_storage_analytics_snapshot
    dataset.save()
  File "/app/src/apps/datasets/models.py", line 72, in save
    self.file_size = self.data_file.size / 1024
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/files.py", line 69, in size
    return self.storage.size(self.name)
  File "/usr/local/lib/python3.8/site-packages/storages/backends/s3boto3.py", line 550, in size
    return self.bucket.Object(self._encode_name(name)).content_length
  File "/usr/local/lib/python3.8/site-packages/boto3/resources/factory.py", line 339, in property_loader
    self.load()
  File "/usr/local/lib/python3.8/site-packages/boto3/resources/factory.py", line 505, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(**params)
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (404) when calling the HeadObject operation: Not Found

The function works well on the test server, but fails on the production server. My quick Google search makes me think the problem is related to an Object in the Bucket that does not exist anymore.

Problem 2:

  • Solved #1464

There also seem to be a problem with the dates shown in the different tabs of the Storage Analytics window. The date shown on the Usage History tab is conflicting with the date shown on the Competitions Usage and the Users Usage tabs.

image image

The same happens on the test server, at least when the function is run manually, the date is the same, but the time is different (the time for the Competitions Usage and the Users Usage tabs is shown to be before the function was run, which is impossible) image

image The correct time is shown on the Usage History tab.