Page MenuHomePhabricator

[services-lib] Add DDB batch operation helpers
ClosedPublic

Authored by bartek on Aug 22 2023, 12:53 AM.
Tags
None
Referenced Files
Unknown Object (File)
Tue, Nov 12, 12:00 PM
Unknown Object (File)
Tue, Nov 12, 11:58 AM
Unknown Object (File)
Wed, Oct 30, 9:12 PM
Unknown Object (File)
Sun, Oct 27, 4:45 PM
Unknown Object (File)
Wed, Oct 23, 2:47 AM
Unknown Object (File)
Oct 21 2024, 1:24 AM
Unknown Object (File)
Oct 19 2024, 4:00 PM
Unknown Object (File)
Oct 19 2024, 3:40 PM
Subscribers

Details

Summary

We use DDB batch operations in several places: reserved usernames write, saving new one-time keys, blob cleanup (soon), reports storage.

DDB BatchWriteItem operation API can be cumbersome - 25 items limit, unprocessed items, throttling etc. Their docs also recommend exponential backoff strategy. I decided to create an API that handles all this overhead for us. See also comments in D8452.

This diff adds some exponential backoff helpers. See child diff for actual batch_write impl.

Test Plan

Tested together with child diff

Diff Detail

Repository
rCOMM Comm
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

bartek held this revision as a draft.
bartek published this revision for review.Aug 22 2023, 1:48 AM

LGTM, depending on how it is used / if there are going to be non-const-creatable fields in the config in the future, we could consider adding a const default:

impl ExponentialBackoffConfig {
  const DEFAULT: Self = Self {
        max_attempts: 8,
        base_duration: std::time::Duration::from_millis(25),
        retry_on_provisioned_capacity_exceeded: true,
      };
}
impl Default for ExponentialBackoffConfig {
    fn default() -> Self {
      Self::DEFAULT
    }
  }

so e.g. a service can just create a global const instead of passing the config in some other way.

LGTM. we should consider adding jitter, though

This revision is now accepted and ready to land.Aug 28 2023, 10:09 AM