Page MenuHomePhabricator

[services][backup] CreateBackup 3/3 - handle chunks and finish
ClosedPublic

Authored by bartek on Jan 9 2023, 4:09 AM.
Tags
None
Referenced Files
Unknown Object (File)
Sat, Nov 30, 1:03 AM
Unknown Object (File)
Sat, Nov 30, 1:03 AM
Unknown Object (File)
Sat, Nov 30, 1:03 AM
Unknown Object (File)
Sat, Nov 30, 1:03 AM
Unknown Object (File)
Sat, Nov 30, 1:03 AM
Unknown Object (File)
Sat, Nov 30, 12:22 AM
Unknown Object (File)
Nov 11 2024, 3:45 AM
Unknown Object (File)
Nov 10 2024, 2:00 PM
Subscribers

Details

Summary

This adds remaining logic for CreateBackup handler: handling data chunks and finishing the request.

Depends on D6198

Test Plan

The whole CreateBackupHandler can be tested for example this way:

cd services && yarn init-local-cloud
yarn run-blob-service-in-sandbox

# in another nix terminal
cd services/backup
RUST_LOG=backup=trace cargo run -- --port 50052 --sandbox --blob-service-url "http://localhost:50053"

Now the endpoint can be called either manually (e.g. BloomRPC) or by yarn run-integration-tests backup - these will fail yet, but the backup should be created, which can be checked by e.g. looking at backup and blob services logs or using aws-cli.

Diff Detail

Repository
rCOMM Comm
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

bartek held this revision as a draft.
bartek published this revision for review.Jan 9 2023, 5:36 AM
services/backup/src/service/handlers/create_backup.rs
116 ↗(On Diff #20671)

Out of scope of this work.

But, I would eventually like to remove should_close_stream. Whether to close a stream should be apparent in the type of messages being received, and communicated through some return value.

Maybe I just need to look into this more, but we seem to inter-mixing the lifetimes of PutRequest, a tonic stream, and the responses in the MyblobService::put method.

There's probably a way to order the logic so that smaller lifetimes can fit into the context of a longer lifetime. Can be future work, but it would be nice to implicitly get correct behavior rather than having to orchestrate invariance across many objects.

This revision is now accepted and ready to land.Jan 11 2023, 9:39 PM

Rebase, changed is_data_mode boolean to enum - this required slight code flow refactors (genral flow remains unchanged)

In D6199, @jon wrote:

But, I would eventually like to remove should_close_stream. Whether to close a stream should be apparent in the type of messages being received, and communicated through some return value.

Maybe I just need to look into this more, but we seem to inter-mixing the lifetimes of PutRequest, a tonic stream, and the responses in the MyblobService::put method.

There's probably a way to order the logic so that smaller lifetimes can fit into the context of a longer lifetime. Can be future work, but it would be nice to implicitly get correct behavior rather than having to orchestrate invariance across many objects.

Agreed, I hope to get rid of this and keep these handlers much simpler when the API gets simplified by merging most inputs together. As for now, this solution "just works" and keeps the code more organized than C++ reactors.
One way to achieve a better flow would be to let it control tonic streams instead of being "event triggered". I managed to make response-streamed PullBackup handler look pretty clean by using async_stream to generate responses: D6241