HomePhabricator
Diffusion Comm 357c69fe954e

[services][backup] PullBackup 5/5 - blob download stream

Description

[services][backup] PullBackup 5/5 - blob download stream

Summary:
This diff leverages both ResponseBuffer and BlobStoredItem to create a generic stream that downloads the item from blob service, buffers the data and sends in chunks not exceeding the gRPC message size. A few notes:

  • First message includes "additional extra info" - fields like attachment holders etc. Rest receive only ID and data chunk
  • The data is buffered before sending, this is the solution 2 from https://phab.comm.dev/D4439#126984
  • More data is not downloaded when the buffer is already filled enough
  • Stream ends when there is no more data

The order of instructions is important here:

  1. Push new data if not full
  2. Finish if empty
  3. Pop if not empty

Depends on D6243

Test Plan:
At this point, the whole service can be tested with integration tests

cd services
yarn reset-local-cloud
nix run .#comm-blob
RUST_LOG=backup=trace cargo run -- --port 50052 --sandbox --blob-service-url "http://localhost:50051"
yarn run-integration-tests backup

Reviewers: tomek, varun, jon, michal

Reviewed By: tomek

Subscribers: ashoat, atul

Differential Revision: https://phab.comm.dev/D6246

Details

Provenance
bartekAuthored on Jan 11 2023, 2:37 PM
Reviewer
tomek
Differential Revision
D6246: [services][backup] PullBackup 5/5 - blob download stream
Parents
rCOMM0a04eeaf590b: [services][backup] PullBackup 4/5 - blob stored item
Branches
Unknown
Tags
Unknown