[CI] Add `backup_build` workflow to Buildkite/GH Actions
ClosedPublic
Actions

Authored by atul on Jun 28 2022, 11:11 PM.

Details

Reviewers

• jacek
tomek
• abosh
• karol

Commits

rCOMM4ddf47c4b62c: [CI] Add `backup_build` workflow to Buildkite/GH Actions

Summary

This was previously handled as part of the "Services CI" workflow which will be deprecated later in this stack. This diff allows us to increase concurrency as we move over to the autoscaling EC2 instances.

Differences on Buildkite side:

We use the actual shell command (cd services && docker-compose build --no-cache backup-server) instead of yarn build-backup-service because the autoscaling instances are intended to be as minimal as possible and don't include node, yarn, etc.

We use the --no-cache flag to force a new complete build on each run. The previous "Services CI" workflow depended heavily on caches, because running 3-4 complete builds would take a long time. Now we can separate these out and take advantage of the concurrency provided by the autoscaling instances.

Test Plan

Buildkite: Copy/pasted from what I have in the Buildkite GUI that has been "active" for the last 2-3 days without issue.

GitHub: Will see what happens when it lands, but fairly confident it should be good to go.

Diff Detail

Repository

rCOMM Comm

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

atul created this revision.Jun 28 2022, 11:11 PM

Herald added subscribers: • adrian, ashoat. · View Herald TranscriptJun 28 2022, 11:11 PM

atul edited the summary of this revision. (Show Details)Jun 28 2022, 11:11 PM

Harbormaster completed remote builds in B10069: Diff 13923.Jun 28 2022, 11:21 PM

atul requested review of this revision.Jun 28 2022, 11:21 PM

atul added a child revision: D4390: [CI] Add `blob_build` workflow to Buildkite/GH Actions.Jun 28 2022, 11:45 PM

On Comm @karol-bisztyga suggested that we should run ./scripts/run_server_image.sh backup

• karol requested changes to this revision.Jun 29 2022, 6:16 AM

• karol added inline comments.

.buildkite/backup_build.yml
2 ↗	(On Diff #13923)	Can we use the bash script instead? ./scripts/run_server_image.sh backup
.github/workflows/backup_build.yml
21 ↗	(On Diff #13923)	Can we use the bash script instead? ./scripts/run_server_image.sh backup

This revision now requires changes to proceed.Jun 29 2022, 6:16 AM

tomek mentioned this in D4390: [CI] Add `blob_build` workflow to Buildkite/GH Actions.Jun 29 2022, 8:20 AM

atul requested review of this revision.Jun 29 2022, 11:39 AM

atul added inline comments.

.buildkite/backup_build.yml
2 ↗	(On Diff #13923)	In addition to `docker-compose build $SERVICE-server`, doesn't this also invoke `docker-compose up $SERVICE-server`? We were previously running `yarn build-all` for the Services CI which was an "alias" for `docker-compose build`, is there a reason we want to change that now?

If we only want to build, it is reasonable to not run the container. @karol-bisztyga what is the reason behind suggesting run_server_image script? Are there any additional benefits?