Page MenuHomePhabricator

[keyserver] Add mysqldump to Node keyserver Docker container
ClosedPublic

Authored by ashoat on Jun 2 2022, 6:56 AM.
Tags
None
Referenced Files
Unknown Object (File)
Sat, Nov 2, 6:20 PM
Unknown Object (File)
Thu, Oct 24, 5:48 AM
Unknown Object (File)
Mon, Oct 21, 7:38 AM
Unknown Object (File)
Sun, Oct 20, 8:24 PM
Unknown Object (File)
Wed, Oct 9, 3:10 AM
Unknown Object (File)
Tue, Oct 8, 8:50 AM
Unknown Object (File)
Tue, Oct 8, 8:50 AM
Unknown Object (File)
Tue, Oct 8, 8:49 AM

Details

Summary

Our backup functionality relies on mysqldump being available on the same image that Node runs on.

Unfortunately, getting mysqldump 5.7 on this image has been quite a challenge.

  1. First thing I tried was to get it from the default apt repos, but it isn't there.
  2. Next I tried to use the same source that the MySQL image uses by inspecting its Docker layers
    • I found it was using an older version of Debian (buster), so I tried downgrading our Node container's Debian version. As it turns out, bullseye (the newer version we're using) doesn't actually have MySQL 5.7... only buster does
    • But it turns out the version for buster only has an x64 build, no ARM (Which is consistent with us needing this line)
    • We could force our Node image to be emulated x64 on Mac but I wasn't excited about that prospect
  3. Next I considered using the default-mysql-client package
    • Found out about it here
    • But based on a comment on that StackOverflow, it looks like it's actually a MariaDB client, and there is some incompatibility with MySQL 5.7
  4. I spent most of the time trying different stuff I found online
  5. I also considered trying to use the mysqldump on the sepatate MySQL container, but this would only be possible from the host, which would either mean setting up Node on the host so we could use our existing code, or replacing our existing code with something like a bash script.
  6. Finally I settled on using mysqldump from MySQL 8
    • To even get this installed separately from the full MySQL server, I had to add Debian's unstable repo
    • Additionally I needed an extra config to disable some new MySQL 8 feature that was crashing when used with MySQL 5.7
    • But I got it working!
Test Plan

First, I configured keyserver/.env as follows:

COMM_MYSQL_DATABASE=commdev
COMM_MYSQL_USER=commdev
COMM_MYSQL_PASSWORD=pass
COMM_JSONCONFIG_facts_landing_url='{"baseDomain":"http://localhost","basePath":"/commlanding/","baseRoutePath":"/commlanding/","https":false}'
COMM_JSONCONFIG_facts_commapp_url='{"baseDomain":"http://localhost","basePath":"/comm/","https":false,"baseRoutePath":"/comm/"}'
COMM_JSONCONFIG_facts_backups='{"enabled":true,"directory":"/home/comm/mysqldump"}'

Then I patched keyserver/docker-compose.yml to load MySQL from a bind-mounted backup (in), and to bind-mount the backups folder (out): https://gist.github.com/Ashoat/ebdd3ae0832a10a950d5f3c8f06f3feb

I also had to create the backups folder on the host machine.

Then I edited keyserver/src/cron/cron.js to increase the frequency of backups to every five minutes: https://gist.github.com/Ashoat/a4a338593b3b2ce1853a264244e6eb0b

Finally, I ran docker-compose down -v && docker-compose build --no-cache && docker-compose up on the host machine

Diff Detail

Repository
rCOMM Comm
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

ashoat requested review of this revision.Jun 2 2022, 7:10 AM

Rename backups folder from /home/comm/mysqldump to /home/comm/backups. (Figured the old name might get confused with the actual mysqldump binary)

But I got it working!

Nice! Appreciate that you included all the steps/issues you ran into in the "Summary" section of the diff.


As an aside, I think there's a task to upgrade from MySQL 5.7 -> 8.x or MariaDB or something that's been open for a while... is that something we should prioritize and handle at the tail end of this work?

https://linear.app/comm/issue/ENG-111/switch-from-mysql-57-to-mariadb-10
But also in https://linear.app/comm/issue/ENG-636/move-phabricator-mysql-57-database-on-ashoats-server-to-newer-mariadb looks like we decided to go with MySQL 8 instead of MariaDB for Phabricator?

keyserver/Dockerfile
39 ↗(On Diff #13295)

Instead of including \n I think I'd slightly prefer something like

RUN echo "[mysqldump]" >> /home/comm/.my.cnf
RUN echo "column-statistics=0" >> /home/comm/.my.cnf

It doesn't make a difference now, but maybe once/if we start adding additional lines to /home/comm/.my.cnf?

This revision is now accepted and ready to land.Jun 2 2022, 10:22 AM

We're still planning on MariaDB for Node. Phabricator uses MySQL because that's what it's targeted at. Not sure about prioritization yet, but agree it should be done soon-ish.

keyserver/Dockerfile
39 ↗(On Diff #13295)

Makes sense