HomePhabricator
Diffusion Comm 8e5753f4104c

[keyserver] run a temporary express health check server during migration for…

Description

[keyserver] run a temporary express health check server during migration for AWS load balancer health checks

Summary:
We had a problem when running our last keyserver migration on Ashoat's keyserver where the migration took much longer than prior migrations. The AWS load balancer's default grace period for the ecs task ran out and
the migration process was cut off in the middle. This diff runs a temporary health check express server during the migration process so AWS can establish the ecs task as healthy

Was originally worried that we would need to fully stop the health check express server, but given that it's only on the master process, I don't believe this is necessary. Additionally, the default close provided function prevents new connections

Test Plan:
Tested this locally by creating a dummy migration that slept for 10 seconds. During this time, keyserver endpoints were unavailable but the health check returned an OK response. Upon completing the migration,
both the health route and keyserver endpoints were available.

Also temporarily modified the health check route to timeout infinitely. Upon finishing the migration, the ongoing health check was terminated and the express server keyserver endpoints was able to be initialized.

Will not land without testing on AWS, ensuring a healthy primary node goes online with a dummy migration running past the default AWS grace period

Reviewers: varun, ashoat

Reviewed By: ashoat

Subscribers: tomek

Differential Revision: https://phab.comm.dev/D13172