Page MenuHomePhabricator

[native] Backup compaction upload
ClosedPublic

Authored by michal on Jan 15 2024, 6:44 AM.
Tags
None
Referenced Files
Unknown Object (File)
Sat, Nov 23, 2:54 PM
Unknown Object (File)
Sat, Nov 23, 1:47 PM
Unknown Object (File)
Sat, Nov 23, 11:58 AM
Unknown Object (File)
Thu, Nov 21, 3:52 AM
Unknown Object (File)
Thu, Nov 21, 3:52 AM
Unknown Object (File)
Oct 22 2024, 1:06 PM
Unknown Object (File)
Oct 22 2024, 9:00 AM
Unknown Object (File)
Oct 22 2024, 5:31 AM
Subscribers

Details

Summary

ENG-6217 : Procedure for handling compaction/log files
ENG-6218 : Upload logic for compactions

General logic for file (compaction and log) upload and compaction specific upload. After enabling the backup handler and each time TRIGGER_BACKUP_FILE_UPLOAD is triggered we want to iterate over all files in backup directory and upload data. A compaction should consist of 3 files: backup-XXX which contains the compaction data (created in C++ database code), backup-123-userkeys which contains the UserKeys in JSON (this will be created from Rust in a later diff) and an optional backup-123-attachments file which contains newline-separated list of attachment hashes (files containing sensitive data will are encrypted). The files names are parsed with regex to get e.g. backupID. The data is uploaded and after a succesfull upload, files are deleted.

Depends on D10630

Test Plan

Create backup-123, backup-123-attachments, backup-123-userkeys files. Enable backup upload. Make sure all data was successfuly uploaded to the backup service and the files were removed.

Diff Detail

Repository
rCOMM Comm
Lint
No Lint Coverage
Unit
No Test Coverage

Event Timeline

native/native_rust_library/Cargo.toml
20 ↗(On Diff #35619)

We already use regex v1, in other parts of our codebase.

native/native_rust_library/src/backup.rs
142–143 ↗(On Diff #35619)

Wasn't sure if non-network related errors should restart the WS connection. I decided that they shouldn't but I welcome any suggestions if there are better ways to handle it.

native/native_rust_library/src/backup.rs
212–213 ↗(On Diff #35619)

Handled in the next diff

native/native_rust_library/src/backup.rs
96 ↗(On Diff #35619)

Even though it's compile-time constant, makes it easier to debug when you made a mistake in the regex string 😛

142–143 ↗(On Diff #35619)

IMO they shouldn't too. Optionally you can add a log message that I/O error occurred

167 ↗(On Diff #35619)

Have you considered extracting contents of this loop to a separate function?
Just business logic, let's keep tokio::fs calls here

unwrap -> expect. Refactored the code into separate functions so it's cleaner.

Some renames and code refactor to make it cleaner

Nice, looks much cleaner than before!

native/native_rust_library/src/backup/upload_handler.rs
217 ↗(On Diff #35788)

Any message here?

This revision is now accepted and ready to land.Jan 19 2024, 7:25 AM