HomePhabricator
Diffusion Comm bdd4a6e23124

[keyserver] Limit mysqldump backup queries to 0.5 MiB

Description

[keyserver] Limit mysqldump backup queries to 0.5 MiB

Summary:
I was seeing a very strange issue where restores of mysqldump backups would fail to correctly restore two specific rows in the notifications table, causing the delivery column to become corrupted.

After extensive testing spanning multiple days, I found two ways to avoid the problem:

  1. This fix, wherein we limit the number of rows in each INSERT.
  2. Deleting the SET FOREIGN_KEY_CHECKS=0 line that gets automatically inserted at the top of the backup file by mysqldump.

I can't tell why the second option works and there's no way to configure mysqldump to prevent it from spitting out that line. We can remove it after-the-fact but that would be more work.

On the other hand, this solution is simple and "just works".

Test Plan:

  1. I patched the change on my production keyserver and generated a backup from there
  2. I downloaded the backup to my laptop and restored it
  3. I then generated a second backup (with the change) on my laptop
  4. Finally I restored the second backup on my laptop, confirming there was no corruption and that everything worked

Reviewers: atul, tomek

Reviewed By: tomek

Subscribers: adrian, abosh

Differential Revision: https://phab.comm.dev/D4642

Details

Provenance
ashoatAuthored on Jul 26 2022, 4:40 AM
Reviewer
tomek
Differential Revision
D4642: [keyserver] Limit mysqldump backup queries to 0.5 MiB
Parents
rCOMM0dc8a1937bd6: [services] Use mutual server listen address
Branches
Unknown
Tags
Unknown