Page MenuHomePhabricator

[services][blob] Handle removing blobs
ClosedPublic

Authored by bartek on Nov 22 2022, 1:03 AM.
Tags
None
Referenced Files
Unknown Object (File)
Mon, Mar 11, 8:26 PM
Unknown Object (File)
Tue, Mar 5, 1:14 AM
Unknown Object (File)
Tue, Mar 5, 1:14 AM
Unknown Object (File)
Tue, Mar 5, 1:14 AM
Unknown Object (File)
Mon, Mar 4, 8:24 PM
Unknown Object (File)
Tue, Feb 27, 6:38 PM
Unknown Object (File)
Feb 25 2024, 2:05 AM
Unknown Object (File)
Feb 19 2024, 6:46 PM
Subscribers

Details

Summary

Implemented the Remove RPC handler.

C++ counterpart

Depends on D5701

Test Plan
  1. Create the blob using e.g. C++ service, or in any other way.
  2. Remove the blob using Rust implementation
  3. Ensure it is deleted in DynamoDB (both tables) and S3

Diff Detail

Repository
rCOMM Comm
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

bartek held this revision as a draft.

This diff is still a draft. Is that intentional?

bartek published this revision for review.Nov 24 2022, 5:26 AM
tomek added inline comments.
services/blob/src/service.rs
146–186 ↗(On Diff #18814)

If deleting reverse index is successful but then deleting the item fails, we would end up in a state where retrying will always fail. This isn't too serious, as we're planning to introduce proper cleanup procedure later, but maybe we should consider making it safer now.

  1. If the request included the hash, we could simply ignore the fact that reverse index can't be found and continue to deleting the item by hash.
  2. Other option is to delete the item first when there's only a single holder referencing it and then delete the index.

The current solution matches what we already have, so maybe it is a good idea to create a followup task for handling deleting in a safer manner.

varun added inline comments.
services/blob/src/service.rs
146–186 ↗(On Diff #18814)

I think we can introduce a “nanny task” later that checks periodically for “orphaned” items and removes them

This revision is now accepted and ready to land.Nov 28 2022, 11:35 PM
services/blob/src/service.rs
146–186 ↗(On Diff #18814)

I admit that this code is problematic as failure anywhere results in an invalid state - tables will go out of sync. IMO removing reverse-index first is the best solution for now, because it makes it inaccessible anymore, even when further deletion (s3 or blob table) goes wrong.

I think we can introduce a “nanny task” later that checks periodically for “orphaned” items and removes them

+1 on this, I think it's more related to ENG-414

This revision was automatically updated to reflect the committed changes.