[backup] Log download endpoint
Summary:
ENG-5330 : Add endpoint for downloading logs
Log download logic. Each request specifies (optional) index from which the fetch should start which allows us to paginate the results. The final message sent by the backup service contains the id from which should the next fetch be started or None of we have already fetched all logs for a given backup.
Depends on D10456
Test Plan:
- Created a 100 logs for 3 backups, half of which were bigger than the dynamodb maximum storage size (so they were stored in blob service)
- Run this code (simplified):
tx.send(DownloadLogsRequest { from_id: None }); 'outer: loop { loop { match rx.next().await { Some(Ok(LogWSResponse::LogDownload { log_id, content, attachments, })) => { // Make sure that content and attachments match the expected values // for this specific logs // ... } Some(Ok(LogWSResponse::LogDownloadFinished { last_log_id })) => { if let Some(last_log_id) = last_log_id { tx.send(DownloadLogsRequest { from_id: Some(last_log_id), }); } else { break 'outer; } } }; } }
- Got all 100 logs for the correct backup with correct data (and in correct order, although final backup service client won't depend on this)
Reviewers: bartek, kamil
Reviewed By: bartek
Subscribers: ashoat, tomek
Differential Revision: https://phab.comm.dev/D10457