Page MenuHomePhabricator

[services] Backup - Add AddAttachment method
ClosedPublic

Authored by karol on May 19 2022, 3:02 AM.
Tags
None
Referenced Files
F2154632: D4080.id13140.diff
Sun, Jun 30, 7:39 PM
F2153816: D4080.id.diff
Sun, Jun 30, 5:59 PM
F2149123: D4080.id13209.diff
Sun, Jun 30, 7:09 AM
F2149122: D4080.id13140.diff
Sun, Jun 30, 7:09 AM
F2149121: D4080.id12924.diff
Sun, Jun 30, 7:09 AM
F2149119: D4080.id13053.diff
Sun, Jun 30, 7:09 AM
F2149102: D4080.id.diff
Sun, Jun 30, 7:09 AM
F2149077: D4080.diff
Sun, Jun 30, 7:05 AM

Details

Summary

Adding AddAttachment method to the backup proto file, plus generating according files with the protobuf.

The idea is, that we first upload a new backup/log, and only after that, we separately upload attachments for it.

An alternative solution would be to inject uploading attachments when uploading a new backup/log but I decided it would be messier, the logic, in the end, would get too complicated, and having the solution I'm proposing here, we end up with more isolated and reusable code that is easier to maintain and understand.

The fields we require in the AddAttachmentRequest:

  • userID - we need this to be ale to fetch the proper backup item.
  • backupID - we need this to be ale to fetch the proper backup item.
  • logID - optional we need this to be able to fetch the proper log item only if the attachment is about to be appended to the log item. If this is empty, it means we're appending to the backup item.
  • dataHash - we need this so we can upload the data to the blob.
  • dataChunk - chunks of the attachment.
Test Plan

None, this change is going to be used later in the stack, it's just a change in the proto file.

Diff Detail

Repository
rCOMM Comm
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

karol edited the test plan for this revision. (Show Details)
karol added reviewers: tomek, ashoat.
karol edited the summary of this revision. (Show Details)
ashoat requested changes to this revision.May 19 2022, 12:52 PM

Sorry to share this, since I suspect it will mean refactoring a lot of your code...

But I don't think attachments should be handled by the backup service directly, unless I'm missing something. They should be handled directly by the blob service. The backup service doesn't need to even backup the attachments directly... it just needs to "hold" the blob (so the blob doesn't get cleaned up).

This revision now requires changes to proceed.May 19 2022, 12:52 PM

So we should upload attachments from the client to the blob directly and also send information to the backup service separately, also from the client, what attachment belongs where?

So we should upload attachments from the client to the blob directly and also send information to the backup service separately, also from the client, what attachment belongs where?

That's right. Worth noting the backup service isn't the only holder for an upload blob.

Here's how I imagine it going:

  1. User uploads attachment to the blob service
    • I forget how we make sure there is a holder at the start, but I remember we discussed this
    • I think we either assign a temporary holder, or we have a rule that we don't delete a zero-holder blob until some time has passed since it was uploaded
  2. User sends the attachment as part of a message
    • This goes to Tunnelbroker, and so Tunnelbroker attachs a holder to the attachment
    • Either we have one holder for each recipient, or just one total holder
    • Tunnelbroker clears its holder(s) once all messages have been delivered to the recipient
  3. Next, both the sender and recipients now have this attachment as part of their backup
    • Each user (sender and recipients) send a log to the backup service referencing this attachment
    • The backup service attaches a separate log for each user
ashoat requested changes to this revision.May 20 2022, 7:06 AM
This revision now requires changes to proceed.May 20 2022, 7:06 AM

I think we can do this without the tunnelbroker layer for now. I think that injecting it later in the tunnelbroker's going to be relatively easy.

Yes I agree! I was just listing everything for context, hoping that seeing the whole thing is helpful

karol edited the summary of this revision. (Show Details)
karol edited the test plan for this revision. (Show Details)

update

ashoat requested changes to this revision.May 24 2022, 6:12 AM

See feedback above, backup service shouldn't handle this

This revision now requires changes to proceed.May 24 2022, 6:12 AM

So should we remove these:

?

All this method (AddAttachment) does is update log/backup items in the backup database with new attachments' values. So the way I see it, we either proceed with what's here or we remove the attachment holders fields from log/backup items. Maybe there's yet another way here?

Re requesting a review for answers.

native/cpp/CommonCpp/grpc/protos/backup.proto
26 ↗(On Diff #13053)

btw this probably should be prular

ashoat requested changes to this revision.May 24 2022, 7:06 AM

Not sure about those lines... I don't have context on what they do. I am pretty sure about the API you're introducing here, though. Please see my thorough comment above to get an idea of how the API here should work.

If you have any further questions, can you please schedule a 1:1 instead of re-requesting review? It feels a bit like we're getting into a bad cycle here... I'm basically copy-pasting my feedback from last time, and I don't think that's good for anybody.

This revision now requires changes to proceed.May 24 2022, 7:06 AM

I'm basically copy-pasting my feedback from last time

Not sure why, honestly. You don't have to keep linking your answer in every following message - I saw it.

Not sure about those lines... I don't have context on what they do.

What context? The entire context is that we store the attachment holders assigned to specific backup/log items. Not much more to say about these two lines. It's pretty simple as I linked just fields of database "entities".

And if we store these holders, then we need to provide them somehow to the backup service.

Each user (sender and recipients) send a log to the backup service referencing this attachment

This is exactly what I'm doing here. How else do you imagine performing at this point? We just have to update items in the database with specific attachment holders, so certain logs have attachment holders assigned accordingly.

I really don't understand what's wrong here for you, this change is really simple but if you feel like blocking this for another day, then fine - let's talk about this in 1v1.

Sorry about my comments above – I failed to actually read the changes in the most recent update to this diff, and I didn't do a good job of reading through @karol-bisztyga's previous comments.

This diff is actually doing something necessary – part 3 of my long comment above.

native/cpp/CommonCpp/grpc/protos/backup.proto
95 ↗(On Diff #13053)

Should we use your new ATTACHMENT_DELIMITER strategy here, and just have a single string with all of the holders? That way we can address ENG-1052 here too

Thanks @karol-bisztyga, and my apologies again for the misunderstanding

This revision is now accepted and ready to land.May 26 2022, 8:04 AM