Page MenuHomePhabricator

[services-lib] Add DDB batch operation helpers
ClosedPublic

Authored by bartek on Aug 22 2023, 12:53 AM.
Tags
None
Referenced Files
Unknown Object (File)
Thu, May 16, 9:49 AM
Unknown Object (File)
Sat, May 4, 1:02 PM
Unknown Object (File)
Tue, Apr 30, 8:30 PM
Unknown Object (File)
Sun, Apr 28, 4:55 PM
Unknown Object (File)
Fri, Apr 26, 2:00 AM
Unknown Object (File)
Apr 16 2024, 10:57 AM
Unknown Object (File)
Apr 16 2024, 8:16 AM
Unknown Object (File)
Apr 15 2024, 3:36 AM
Subscribers

Details

Summary

We use DDB batch operations in several places: reserved usernames write, saving new one-time keys, blob cleanup (soon), reports storage.

DDB BatchWriteItem operation API can be cumbersome - 25 items limit, unprocessed items, throttling etc. Their docs also recommend exponential backoff strategy. I decided to create an API that handles all this overhead for us. See also comments in D8452.

This diff adds some exponential backoff helpers. See child diff for actual batch_write impl.

Test Plan

Tested together with child diff

Diff Detail

Repository
rCOMM Comm
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

bartek held this revision as a draft.
bartek published this revision for review.Aug 22 2023, 1:48 AM

LGTM, depending on how it is used / if there are going to be non-const-creatable fields in the config in the future, we could consider adding a const default:

impl ExponentialBackoffConfig {
  const DEFAULT: Self = Self {
        max_attempts: 8,
        base_duration: std::time::Duration::from_millis(25),
        retry_on_provisioned_capacity_exceeded: true,
      };
}
impl Default for ExponentialBackoffConfig {
    fn default() -> Self {
      Self::DEFAULT
    }
  }

so e.g. a service can just create a global const instead of passing the config in some other way.

LGTM. we should consider adding jitter, though

This revision is now accepted and ready to land.Aug 28 2023, 10:09 AM