Page MenuHomePhabricator

[services-lib] Add DDB batch operation helpers
ClosedPublic

Authored by bartek on Aug 22 2023, 12:53 AM.
Tags
None
Referenced Files
Unknown Object (File)
Mon, Nov 25, 7:09 PM
Unknown Object (File)
Mon, Nov 25, 7:03 PM
Unknown Object (File)
Mon, Nov 25, 7:02 PM
Unknown Object (File)
Mon, Nov 25, 12:28 PM
Unknown Object (File)
Nov 22 2024, 9:32 AM
Unknown Object (File)
Nov 22 2024, 4:45 AM
Unknown Object (File)
Nov 12 2024, 12:00 PM
Unknown Object (File)
Nov 12 2024, 11:58 AM
Subscribers

Details

Summary

We use DDB batch operations in several places: reserved usernames write, saving new one-time keys, blob cleanup (soon), reports storage.

DDB BatchWriteItem operation API can be cumbersome - 25 items limit, unprocessed items, throttling etc. Their docs also recommend exponential backoff strategy. I decided to create an API that handles all this overhead for us. See also comments in D8452.

This diff adds some exponential backoff helpers. See child diff for actual batch_write impl.

Test Plan

Tested together with child diff

Diff Detail

Repository
rCOMM Comm
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

bartek held this revision as a draft.
bartek published this revision for review.Aug 22 2023, 1:48 AM

LGTM, depending on how it is used / if there are going to be non-const-creatable fields in the config in the future, we could consider adding a const default:

impl ExponentialBackoffConfig {
  const DEFAULT: Self = Self {
        max_attempts: 8,
        base_duration: std::time::Duration::from_millis(25),
        retry_on_provisioned_capacity_exceeded: true,
      };
}
impl Default for ExponentialBackoffConfig {
    fn default() -> Self {
      Self::DEFAULT
    }
  }

so e.g. a service can just create a global const instead of passing the config in some other way.

LGTM. we should consider adding jitter, though

This revision is now accepted and ready to land.Aug 28 2023, 10:09 AM