services/tunnelbroker/src/Database/DatabaseManager.cpp
212–222 ↗	(On Diff #13367)	This is inefficient - we have a lot of requests to the db. Is there a way to delete all the items that match a condition? (something like `delete from ... where ...` in SQL)

This revision now requires changes to proceed.Jun 7 2022, 9:52 AM

• max planned changes to this revision.Jun 8 2022, 2:46 PM

• max mentioned this in D4373: [services] Inner function to write to DynamoDB by batch.Jun 28 2022, 1:50 AM

Switching to using batch deletion instead of one by one.

• max added a parent revision: D4373: [services] Inner function to write to DynamoDB by batch.Jun 28 2022, 1:54 AM

Harbormaster failed remote builds in B10029: Diff 13868!Jun 28 2022, 1:54 AM

• max marked an inline comment as done.Jun 28 2022, 2:00 AM

• max added inline comments.

services/tunnelbroker/src/Database/DatabaseManager.cpp
212–222 ↗	(On Diff #13367)	This is inefficient - we have a lot of requests to the db. Is there a way to delete all the items that match a condition? (something like `delete from ... where ...` in SQL) Unfortunately, we can't delete items from DynamoDB by anything except the primary key, which is a messageID in our case. We need first to fetch these IDs and then remove items by them. But we can use a batch write to remove the items by batch. It's more efficient to put and delete items by batch than one by one. I've updated this diff to use the batch write inner function from D4373.

tomek requested changes to this revision.Jun 28 2022, 11:06 AM

tomek added inline comments.

services/tunnelbroker/src/Database/DatabaseManager.cpp
214 ↗	(On Diff #13868)	We use `uint64_t createdAt` in `MessageItem` - can we use it also here?
212–222 ↗	(On Diff #13367)	According to https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_DeleteItem.html Deletes a single item in a table by primary key. But also You can perform a conditional delete operation that deletes the item if it exists, or if it has an expected attribute value. Could you check if `conditional delete` can be used to avoid fetching all the messages? I think we could specify partition key and condition for the database to delete all the items that have this key and match condition. Fetching and filtering on services side is really inefficient. If it isn't possible, what should we change about our data model to allow it?

This revision now requires changes to proceed.Jun 28 2022, 11:06 AM

Rebase on the parent changes. Change checkpoint variable type to uint64_t.

• max planned changes to this revision.Jun 28 2022, 12:40 PM

• max added inline comments.

services/tunnelbroker/src/Database/DatabaseManager.cpp
214 ↗	(On Diff #13868)	We use `uint64_t createdAt` in `MessageItem` - can we use it also here? Yes, sure. It was changed.

Harbormaster completed remote builds in B10050: Diff 13890.Jun 28 2022, 12:49 PM

• max edited the summary of this revision. (Show Details)Jun 28 2022, 2:03 PM

This diff is canceled due to the architecture changes in ENG-1158 (comment):
We've moved from time-based checkpointing to the messageID-based and removing by a batch in D4476 and ENG-1361.