HomePhabricator
Diffusion Comm df7964047d75

[backup] Log download endpoint

Description

[backup] Log download endpoint

Summary:
ENG-5330 : Add endpoint for downloading logs

Log download logic. Each request specifies (optional) index from which the fetch should start which allows us to paginate the results. The final message sent by the backup service contains the id from which should the next fetch be started or None of we have already fetched all logs for a given backup.

Depends on D10456

Test Plan:

  • Created a 100 logs for 3 backups, half of which were bigger than the dynamodb maximum storage size (so they were stored in blob service)
  • Run this code (simplified):
tx.send(DownloadLogsRequest { from_id: None });

'outer: loop {
  loop {
    match rx.next().await {
      Some(Ok(LogWSResponse::LogDownload {
        log_id,
        content,
        attachments,
      })) => {
        // Make sure that content and attachments match the expected values
        // for this specific logs
        // ...
      }
      Some(Ok(LogWSResponse::LogDownloadFinished { last_log_id })) => {
        if let Some(last_log_id) = last_log_id {
          tx.send(DownloadLogsRequest {
            from_id: Some(last_log_id),
          });
        } else {
          break 'outer;
        }
      }
    };
  }
}
  • Got all 100 logs for the correct backup with correct data (and in correct order, although final backup service client won't depend on this)

Reviewers: bartek, kamil

Reviewed By: bartek

Subscribers: ashoat, tomek

Differential Revision: https://phab.comm.dev/D10457

Details

Provenance
Michal Gniadek <michal.gniadek@swmansion.com>Authored on Dec 22 2023, 1:20 AM
Reviewer
bartek
Differential Revision
D10457: [backup] Log download endpoint
Parents
rCOMMca5496d07ea0: [backup] Log upload endpoint
Branches
Unknown
Tags
Unknown

Event Timeline

Michal Gniadek <michal.gniadek@swmansion.com> committed rCOMMdf7964047d75: [backup] Log download endpoint (authored by Michal Gniadek <michal.gniadek@swmansion.com>).Jan 8 2024, 7:19 AM