[services-lib] Add BatchWriteItem utility function
Summary:
From parent diff description:
We use DDB batch operations in several places: reserved usernames write, saving new one-time keys, blob cleanup (soon), reports storage.
DDB BatchWriteItem operation API can be cumbersome - 25 items limit, unprocessed items, throttling etc. Their docs also recommend exponential backoff strategy. I decided to create an API that handles all this overhead for us. See also comments in D8452.
This diff uses some helpers from parent diff (D8894).
Depends on D8894
Test Plan:
- Preparation
On staging, created two tables: one with on-demand capacity, second one with provisioned capacity, with 1 write capacity unit. Also created one on localstack. The tables were simple one column tables.
- Tests
Did the following:- Generated ~2k (not a multiple of 25) UUIDs (random strings). First test was to save them all in one batch.
- Repeated, but split into chunks of ~400 items (413 IIRC). Spawned a spearate thread for each chunk and ran them concurrently.
- Also, during report service development, imported over 3K reports (100MB) using this function. Each 25-item batch was sent in a separate thread. This was done on localstack.
- Observations
- Localstack doesn't do any throttling so all items were written.
- Similiar for on-demand capacity table. In fact it is able to throttle, but the limit is way too high for my tests.
- With provisioned capacity table, I was able to observe several ProvisionedThroughputExceeded errors. Exponential backoff helped with them and after a few retries all rows were able to be written. However, for large concurrent batch, I had to increase the base delay to give DDB time to get up.
- Example log: https://pastebin.com/embed_js/GmAkXHjk
- Never was able to "partially" succeed - i.e. the response was successful, but unprocessed_items was not empty.
Reviewers: varun, michal, jon
Reviewed By: varun, michal
Subscribers: ashoat, tomek
Differential Revision: https://phab.comm.dev/D8895