HomePhabricator
Diffusion Comm a955e65db94e

[services-lib] Add BatchWriteItem utility function

Description

[services-lib] Add BatchWriteItem utility function

Summary:
From parent diff description:

We use DDB batch operations in several places: reserved usernames write, saving new one-time keys, blob cleanup (soon), reports storage.

DDB BatchWriteItem operation API can be cumbersome - 25 items limit, unprocessed items, throttling etc. Their docs also recommend exponential backoff strategy. I decided to create an API that handles all this overhead for us. See also comments in D8452.

This diff uses some helpers from parent diff (D8894).

Depends on D8894

Test Plan:

  1. Preparation

    On staging, created two tables: one with on-demand capacity, second one with provisioned capacity, with 1 write capacity unit. Also created one on localstack. The tables were simple one column tables.
  1. Tests

    Did the following:
    • Generated ~2k (not a multiple of 25) UUIDs (random strings). First test was to save them all in one batch.
    • Repeated, but split into chunks of ~400 items (413 IIRC). Spawned a spearate thread for each chunk and ran them concurrently.
    • Also, during report service development, imported over 3K reports (100MB) using this function. Each 25-item batch was sent in a separate thread. This was done on localstack.
  1. Observations
    • Localstack doesn't do any throttling so all items were written.
    • Similiar for on-demand capacity table. In fact it is able to throttle, but the limit is way too high for my tests.
    • With provisioned capacity table, I was able to observe several ProvisionedThroughputExceeded errors. Exponential backoff helped with them and after a few retries all rows were able to be written. However, for large concurrent batch, I had to increase the base delay to give DDB time to get up.
    • Never was able to "partially" succeed - i.e. the response was successful, but unprocessed_items was not empty.

Reviewers: varun, michal, jon

Reviewed By: varun, michal

Subscribers: ashoat, tomek

Differential Revision: https://phab.comm.dev/D8895

Details

Provenance
bartekAuthored on Aug 19 2023, 4:38 AM
Reviewer
varun
Differential Revision
D8895: [services-lib] Add BatchWriteItem utility function
Parents
rCOMM902a52421098: [services-lib] Add DDB batch operation helpers
Branches
Unknown
Tags
Unknown