HomePhabricator
Diffusion Comm 775fafa1a75a

[services][blob] Upload 1/2 - Introduce Put handlers

Description

[services][blob] Upload 1/2 - Introduce Put handlers

Summary:
Support for uploading blobs with Put RPC call. This is the most complicated part of the service as it is stateful.

This is part 1 of 2

I implemented it slightly differently from the C++ counterpart. I created a separate PutHandler struct to organize the code. A few notes on how it works:

  • In each input stream message, client can send either holder, blob hash or data chunk.
  • Put handler starts with a PutAction::None state and waits for both holder and hash to be provided (in any order, but each only once).
  • When both of them exist, next state is determined:
    • If blob hash already exists, the AssignHolder action is scheduled and the input stream is closed by the server.
    • Otherwise, the UploadNewBlob action is initialized and handler is waiting for data chunks

Here is where this diff ends, further actions are in the next diff

  • (next diff) Messages containng data chunks can only be sent when handler is in the UploadNewBlob state. Chunks are uploaded to S3 if big enough. The Multipart upload session is initialized when first chunk is sent.
  • (next diff) After input stream is closed, the finish() method is invoked, which consumes the handler and performs action depending on the enum value:
    • If no action, just return
    • For AssignHolder - just add a new row to the reverse_index table
    • In case of UploadNewBlob - finishes the S3 upload and adds rows to both DB tables

Depends on D5702

Test Plan:
Now all gRPC handlers are implemented so running integration tests against this service is possible.
In constants set port to 50053 and add the -test suffix to both table names, run the service, then run yarn run-integration-tests blob.

Reviewers: varun, jon, tomek, marcin, ashoat

Reviewed By: varun, tomek, ashoat

Subscribers: ashoat, atul

Differential Revision: https://phab.comm.dev/D5703

Details

Provenance
bartekAuthored on Nov 28 2022, 1:25 AM
Reviewer
varun
Differential Revision
D5703: [services][blob] Upload 1/2 - Introduce Put handlers
Parents
rCOMM45e12d147022: Unpack draft object to key and text in commCoreModule updateDraft method…
Branches
Unknown
Tags
Unknown