[server] Move to rereadable-stream for caching mysqldump result
Summary:
During backup of the production MySQL squadcal database, we call mysqldump and cache its result in Node.js memory. We then try writing it, and if the write fails we clear up some disk space and try again.
We were previously using stream-cache to cache the mysqldump results. However, I started noticing issues where the write would silently fail due to running out of disk space. In this case the partially written file should be deleted (and then a cleanup and a retry should occur), but because the failure was "silent" the file was being left in place. This gradually led to the backup folder running out of space for Phabricator backups (which get stored in the same folder).
The issue is that the Phabricator backups are larger than the production database backups. When there's not space for a full squadcal backup and it eats up all of the remaining space, the next Phabricator backup (assuming it's bigger than the last Phabricator backup) sometimes doesn't have enough space for a new backup even after deleting the previous one.
More details about this investigation can be found in D561.
Test Plan:
- First I ran the backup logic locally and made sure it worked. I also tracked memory usage of the server to make sure there weren't any memory leaks.
- Next I'm actually going to deploy this diff directly to prod. I can't test the running-out-of-disk-space scenario locally, but need to make sure it works well on prod. For that reason I'm going to push this diff without review, which we should normally avoid doing.
Reviewers: palys-swm
Subscribers: KatPo, zrebcu411, Adrian, atul, subnub
Differential Revision: https://phabricator.ashoat.com/D621