[commtest] Add dockerfile and entrypoint script
ClosedPublic
Actions

Authored by bartek on Oct 17 2023, 5:50 AM.

Details

Reviewers

Commits

Summary

Created a bash script that initializes the localstack for tests, and then runs commtest integration tests. This script is the main entrypoint for the docker container that runs the tests on CI.
Created a Dockerfile for commtest that install necessary dependencies and copies necessary files

Depends on D9510

Test Plan

Tested together with the next diff. It can work without it but setting up the environment manually is too cumbersome.

Repository

rCOMM Comm

Lint

No Lint Coverage

Unit

No Test Coverage

bartek held this revision as a draft.

bartek published this revision for review.Oct 17 2023, 6:06 AM

bartek added 1 blocking reviewer(s): kamil.

bartek added inline comments.

services/commtest/run-tests-ci.sh
22–32	I'll fix this before landing, this is just to show how it works: Unix glob patterns work Single test suite names work Other tests will be uncommented once I'll rebase on all ongoing changes cc @kamil - Is this way of running Tunnelbroker tests okay?

services/commtest/run-tests-ci.sh
22–24	I'm a bit worried about CI performance, which is/was an issue. Are these going to be running sequentially? How much work would it be to make them run in parallel? I'm guessing at least some changes to tests because of id reuse between test cases.

services/commtest/run-tests-ci.sh
22–24	Most of the tests need to run in sequence because of their design, specifically Tunnelbroker tests, for them I'll also add the `--test-threads=1` flag. Glob pattern (as for blob) helps a bit - it defers test scheduling to cargo - and I'll do it for backup as well. If all tests could run in parallel, I'd just do `cargo test` once, without specifying `--test` flag separately for each

kamil added inline comments.

services/commtest/run-tests-ci.sh
22–24	If we want to make Tunnelbroker tests run in parallel we could implement dynamically generating `olm` accounts, now they need to be sequential because of using the same device IDs for different tests (message order is not deterministic)
22–32	Looks good, the best will be glob pattern, to avoid modifying this file each time.

This revision is now accepted and ready to land.Oct 18 2023, 3:50 AM

Rebased on latest master.

A few changes:

Installed aws-cli to have more control over the test environment.
- Added code that force-deletes secretmanager secrets because latest localstack doesn't always do it correctly when doing it with terraform.
Added possibility to pass flags to specific tests, specifically --test-threads=1 for tunnelbroker tests
Used wildcard for blob, backup, tunnelbroker tests
Enabled identity tests. Some of them appeared to be flaky so forced execution order that passes. Linear task: ENG-5298.
Added failure counting instead of early exit on first failure.