diff --git a/docs/dev_services.md b/docs/dev_services.md index 09ed02c74..adf6f31f7 100644 --- a/docs/dev_services.md +++ b/docs/dev_services.md @@ -1,105 +1,109 @@ # Requirements At the moment, our services can be built and run on Linux and macOS via Docker. Unfortunately, Windows is not supported at this time. You’ll ideally want a machine with at least 16 GiB of RAM because running a Docker container can consume up to 4 GiB of RAM. We use Ubuntu as the base Docker image for services. # Prerequisites ## Docker To build and run the services you need to install [Docker](https://docs.docker.com/desktop/) and [Docker Compose](https://docs.docker.com/compose/install) on your system. ## Node We use the `yarn` package manager to install dependencies and run scripts. Installation instructions can be found in the [dev_environment doc](https://github.com/CommE2E/comm/blob/master/docs/dev_environment.md#node). ## AWS Some of our services access AWS resources via the AWS C++ SDK. To access these resources, you'll need to configure the `~/.aws/credentials` and `~/.aws/config` files correctly on your host machine. Instructions for setting these configuration files can be found [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html). We recommend running `aws configure`, which will prompt you for the necessary configuration values. ## Terraform We use [Terraform](https://www.terraform.io/) to create and manage our AWS resources. Installation instructions can be found [here](https://www.terraform.io/downloads). ## RabbitMQ (Tunnelbroker only) [RabbitMQ](https://www.rabbitmq.com/) is an open-source message broker service. We use RabbitMQ in Tunnelbroker to facilitate communication between devices and keyservers. We use the secure AMQPS protocol to connect to RabbitMQ instances hosted on AWS. In order to access and manage RabbitMQ instances, you'll need credentials and the [proper permissions](https://www.rabbitmq.com/access-control.html). You can add new users or edit permissions for existing ones through the [RabbitMQ Management plugin](https://www.rabbitmq.com/management.html). Alternatively, you can manage credentials and permissions from the `rabbitmqctl` CLI. For example, to add a new user you can run the following command: ``` rabbitmqctl add_user {username} ``` You'll need to create a Tunnelbroker-specific configuration file. ``` vim services/tunnelbroker/tunnelbroker.ini ``` Provide a unique ID for each running instance of Tunnelbroker and a RabbitMQ URI in accordance with this [specification](https://www.rabbitmq.com/uri-spec.html). ``` [tunnelbroker] instance-id = tunnelbroker1 [amqp] uri = amqp://guest:guest@0.0.0.0/vhost ``` # Building and running `services/package.json` provides useful scripts to build and run services. The `run` scripts will automatically build the services if necessary and run them. You can find the full list of scripts [here](https://github.com/CommE2E/comm/blob/master/services/package.json) in the `scripts` section. # Developing and debugging ## Visual Studio Code If you are using Visual Studio Code as your code editor you can [attach to a Docker container](https://code.visualstudio.com/docs/remote/attach-container) and develop inside it. ## Sandbox environment for services You can run the Comm services locally in a sandbox environment for development and testing purposes. The sandbox uses a [local cloud stack](https://localstack.cloud/) that includes DynamoDB and S3 running locally in Docker containers. The sandbox also includes a [RabbitMQ](https://www.rabbitmq.com/) Docker container, which is required by Tunnelbroker. ### Configuration changes in the sandbox In your sandbox, services will connect to a local cloud stack, ignoring the `~/.aws` connection settings. The `-test` suffix is applied for all DynamoDB table names in this mode. Tunnelbroker will use the `services/tunnelbroker/tunnelbroker-dev.ini` configuration file and connect to a local instance of the RabbitMQ server. The log level in this mode is increased from ERROR to INFO. ### Running services in the sandbox First, you need to initialize the local cloud using the following command from the the `services` directory: ``` yarn init-local-cloud ``` This will start the LocalStack Docker image and initialize required resources, including DynamoDB tables and S3 buckets, using the Terraform scripts located in `services/terraform`. To start a certain service in the sandbox you can run the following command: ``` yarn run-[service-name]-service-in-sandbox ``` For example, for Tunnelbroker the command will look like this: ``` yarn run-tunnelbroker-service-in-sandbox ``` You can also run all services at once in the sandbox using the command below: ``` yarn run-all-services-in-sandbox ``` + +### Rebuilding the base image + +If you ever wish to rebuild the base image, you should get a tool named [buildx](https://github.com/docker/buildx). It should be attached with the Docker desktop app on the macOS, but if you use Linux, you will probably need to install it manually. For the installation instructions, please go [here](https://github.com/docker/buildx#installing). diff --git a/services/backup/Dockerfile b/services/backup/Dockerfile index b1626b716..364757143 100644 --- a/services/backup/Dockerfile +++ b/services/backup/Dockerfile @@ -1,23 +1,23 @@ -FROM commapp/services-base:1.1 +FROM commapp/services-base:1.2 RUN apt-get update && \ apt-get install -y uuid-dev && \ rm -rf /var/lib/apt/lists/* ARG COMM_TEST_SERVICES ARG COMM_SERVICES_SANDBOX ENV COMM_TEST_SERVICES=${COMM_TEST_SERVICES} ENV COMM_SERVICES_SANDBOX=${COMM_SERVICES_SANDBOX} WORKDIR /transferred COPY native/cpp/CommonCpp/grpc/protos/backup.proto native/cpp/CommonCpp/grpc/protos/blob.proto protos/ COPY services/lib/cmake-components cmake-components COPY services/lib/docker/ scripts/ COPY services/backup/ . COPY services/lib/src/* src/ RUN scripts/build_service.sh CMD if [ "$COMM_TEST_SERVICES" -eq 1 ]; then scripts/run_tests.sh; else scripts/run_service.sh; fi diff --git a/services/blob/Dockerfile b/services/blob/Dockerfile index 6d9eb1aff..50322725a 100644 --- a/services/blob/Dockerfile +++ b/services/blob/Dockerfile @@ -1,23 +1,23 @@ -FROM commapp/services-base:1.1 +FROM commapp/services-base:1.2 RUN apt-get update && \ apt-get install -y uuid-dev && \ rm -rf /var/lib/apt/lists/* ARG COMM_TEST_SERVICES ARG COMM_SERVICES_SANDBOX ENV COMM_TEST_SERVICES=${COMM_TEST_SERVICES} ENV COMM_SERVICES_SANDBOX=${COMM_SERVICES_SANDBOX} WORKDIR /transferred COPY native/cpp/CommonCpp/grpc/protos/blob.proto protos/blob.proto COPY services/lib/cmake-components cmake-components COPY services/lib/docker/ scripts/ COPY services/blob/ . COPY services/lib/src/* src/ RUN scripts/build_service.sh CMD if [ "$COMM_TEST_SERVICES" -eq 1 ]; then scripts/run_tests.sh; else scripts/run_service.sh; fi diff --git a/services/scripts/build_base_image.sh b/services/scripts/build_base_image.sh index 1e391a563..82925bfee 100755 --- a/services/scripts/build_base_image.sh +++ b/services/scripts/build_base_image.sh @@ -1,11 +1,30 @@ #!/usr/bin/env bash set -e if [[ "$#" -gt 1 ]]; then echo "usage: $0 [TAG]" exit 1 fi -tag=${1:-"1.1"} -docker build -t commapp/services-base:"${tag}" base-image + +tag=${1:-"1.2"} + +BUILDER_NAME="COMM_BUILDER" + +BUILDER_PRESENT=$(docker buildx inspect "$BUILDER_NAME" 2> /dev/null || echo "") + +if [[ -z "${BUILDER_PRESENT}" ]]; then + echo "builder not found, creating builder $BUILDER_NAME"; + docker buildx create --name "$BUILDER_NAME" +fi; + +docker buildx use "$BUILDER_NAME" + +# use --push to automatically push this image to the hub +docker buildx build \ + --tag commapp/services-base:"${tag}" \ + -o type=image \ + --platform=linux/arm64,linux/amd64 \ + # --push \ + base-image/. diff --git a/services/tunnelbroker/Dockerfile b/services/tunnelbroker/Dockerfile index fa7efd90c..d239cf941 100644 --- a/services/tunnelbroker/Dockerfile +++ b/services/tunnelbroker/Dockerfile @@ -1,43 +1,43 @@ -FROM commapp/services-base:1.1 +FROM commapp/services-base:1.2 ARG MAKE_JOBS=4 ENV MAKEFLAGS="-j${MAKE_JOBS}" WORKDIR /transferred/scripts # Install Curl RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -y curl # Install SDKs COPY services/tunnelbroker/docker/install_amqp_cpp.sh . RUN ./install_amqp_cpp.sh COPY services/tunnelbroker/docker/install_cryptopp.sh . RUN ./install_cryptopp.sh COPY services/tunnelbroker/docker/install_libuv.sh . RUN ./install_libuv.sh RUN curl https://sh.rustup.rs -sSf | sh -s -- -y COPY services/tunnelbroker/docker/install_corrosion.sh . RUN ./install_corrosion.sh ARG COMM_TEST_SERVICES ARG COMM_SERVICES_SANDBOX ENV COMM_TEST_SERVICES=${COMM_TEST_SERVICES} ENV COMM_SERVICES_SANDBOX=${COMM_SERVICES_SANDBOX} WORKDIR /transferred COPY native/cpp/CommonCpp/grpc/protos/tunnelbroker.proto protos/tunnelbroker.proto COPY services/lib/cmake-components cmake-components COPY services/lib/docker/ scripts/ COPY services/tunnelbroker/docker/* docker/ COPY services/tunnelbroker/ . COPY services/lib/src/* src/ RUN scripts/build_service.sh CMD if [ "$COMM_TEST_SERVICES" -eq 1 ]; then scripts/run_service.sh; else scripts/run_service.sh; fi