diff --git a/docs/dev_services.md b/docs/dev_services.md index 47e263ef9..50258da7b 100644 --- a/docs/dev_services.md +++ b/docs/dev_services.md @@ -1,101 +1,101 @@ # Requirements At the moment, our services can be built and run on Linux and macOS via Docker. Unfortunately, Windows is not supported at this time. You’ll ideally want a machine with at least 16 GiB of RAM because running a Docker container can consume up to 4 GiB of RAM. We use Ubuntu as the base Docker image for services. # Prerequisites ## Docker To build and run the services you need to install [Docker](https://docs.docker.com/desktop/) and [Docker Compose](https://docs.docker.com/compose/install) on your system. ## Node We use the `yarn` package manager to install dependencies and run scripts. Installation instructions can be found in the [dev_environment doc](https://github.com/CommE2E/comm/blob/master/docs/dev_environment.md#node). ## AWS Some of our services access AWS resources via the AWS C++ SDK. To access these resources, you'll need to configure the `~/.aws/credentials` and `~/.aws/config` files correctly on your host machine. Instructions for setting these configuration files can be found [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html). We recommend running `aws configure`, which will prompt you for the necessary configuration values. ## RabbitMQ (Tunnelbroker only) [RabbitMQ](https://www.rabbitmq.com/) is an open-source message broker service. We use RabbitMQ in Tunnelbroker to facilitate communication between devices and keyservers. We use the secure AMQPS protocol to connect to RabbitMQ instances hosted on AWS. In order to access and manage RabbitMQ instances, you'll need credentials and the [proper permissions](https://www.rabbitmq.com/access-control.html). You can add new users or edit permissions for existing ones through the [RabbitMQ Management plugin](https://www.rabbitmq.com/management.html). Alternatively, you can manage credentials and permissions from the `rabbitmqctl` CLI. For example, to add a new user you can run the following command: ``` rabbitmqctl add_user {username} ``` You'll need to create a Tunnelbroker-specific configuration file. ``` vim services/tunnelbroker/tunnelbroker.ini ``` Provide a unique ID for each running instance of Tunnelbroker and a RabbitMQ URI in accordance with this [specification](https://www.rabbitmq.com/uri-spec.html). ``` [tunnelbroker] instance-id = tunnelbroker1 [amqp] uri = amqp://guest:guest@0.0.0.0/vhost ``` # Building and running `services/package.json` provides useful scripts to build and run services. The `run` scripts will automatically build the services if necessary and run them. You can find the full list of scripts [here](https://github.com/CommE2E/comm/blob/master/services/package.json) in the `scripts` section. # Developing and debugging ## Visual Studio Code If you are using Visual Studio Code as your code editor you can [attach to a Docker container](https://code.visualstudio.com/docs/remote/attach-container) and develop inside it. ## Sandbox environment for services You can run the Comm services locally in a sandbox environment for development and testing purposes. The sandbox uses a [local cloud stack](https://localstack.cloud/) that includes DynamoDB and S3 running locally in Docker containers. The sandbox also includes a [RabbitMQ](https://www.rabbitmq.com/) Docker container, which is required by Tunnelbroker. ### Configuration changes in the sandbox In your sandbox, services will connect to a local cloud stack, ignoring the `~/.aws` connection settings. The `-test` suffix is applied for all DynamoDB table names in this mode. Tunnelbroker will use the `services/tunnelbroker/tunnelbroker-dev.ini` configuration file and connect to a local instance of the RabbitMQ server. The log level in this mode is increased from ERROR to INFO. ### Running services in the sandbox First, you need to initialize the local cloud using the following command from the the `services` directory: ``` yarn init-local-cloud ``` This will start the LocalStack Docker image and initialize required resources, including DynamoDB tables and S3 buckets, using [Terraform](https://www.terraform.io/) scripts located in `services/terraform`. To start a certain service in the sandbox you can run the following command: ``` -yarn run-[service-name]-service-dev-mode +yarn run-[service-name]-service-in-sandbox ``` For example, for Tunnelbroker the command will look like this: ``` -yarn run-tunnelbroker-service-dev-mode +yarn run-tunnelbroker-service-in-sandbox ``` You can also run all services at once in the sandbox using the command below: ``` -yarn run-all-services-dev-mode +yarn run-all-services-in-sandbox ``` diff --git a/services/backup/Dockerfile b/services/backup/Dockerfile index cf48baf7c..b1626b716 100644 --- a/services/backup/Dockerfile +++ b/services/backup/Dockerfile @@ -1,23 +1,23 @@ FROM commapp/services-base:1.1 RUN apt-get update && \ apt-get install -y uuid-dev && \ rm -rf /var/lib/apt/lists/* ARG COMM_TEST_SERVICES -ARG COMM_SERVICES_DEV_MODE +ARG COMM_SERVICES_SANDBOX ENV COMM_TEST_SERVICES=${COMM_TEST_SERVICES} -ENV COMM_SERVICES_DEV_MODE=${COMM_SERVICES_DEV_MODE} +ENV COMM_SERVICES_SANDBOX=${COMM_SERVICES_SANDBOX} WORKDIR /transferred COPY native/cpp/CommonCpp/grpc/protos/backup.proto native/cpp/CommonCpp/grpc/protos/blob.proto protos/ COPY services/lib/cmake-components cmake-components COPY services/lib/docker/ scripts/ COPY services/backup/ . COPY services/lib/src/* src/ RUN scripts/build_service.sh CMD if [ "$COMM_TEST_SERVICES" -eq 1 ]; then scripts/run_tests.sh; else scripts/run_service.sh; fi diff --git a/services/blob/Dockerfile b/services/blob/Dockerfile index 427ac2d0a..6d9eb1aff 100644 --- a/services/blob/Dockerfile +++ b/services/blob/Dockerfile @@ -1,23 +1,23 @@ FROM commapp/services-base:1.1 RUN apt-get update && \ apt-get install -y uuid-dev && \ rm -rf /var/lib/apt/lists/* ARG COMM_TEST_SERVICES -ARG COMM_SERVICES_DEV_MODE +ARG COMM_SERVICES_SANDBOX ENV COMM_TEST_SERVICES=${COMM_TEST_SERVICES} -ENV COMM_SERVICES_DEV_MODE=${COMM_SERVICES_DEV_MODE} +ENV COMM_SERVICES_SANDBOX=${COMM_SERVICES_SANDBOX} WORKDIR /transferred COPY native/cpp/CommonCpp/grpc/protos/blob.proto protos/blob.proto COPY services/lib/cmake-components cmake-components COPY services/lib/docker/ scripts/ COPY services/blob/ . COPY services/lib/src/* src/ RUN scripts/build_service.sh CMD if [ "$COMM_TEST_SERVICES" -eq 1 ]; then scripts/run_tests.sh; else scripts/run_service.sh; fi diff --git a/services/blob/src/S3Tools.cpp b/services/blob/src/S3Tools.cpp index cd6ac8869..00512582d 100644 --- a/services/blob/src/S3Tools.cpp +++ b/services/blob/src/S3Tools.cpp @@ -1,47 +1,47 @@ #include "S3Tools.h" #include "Constants.h" #include "GlobalConstants.h" #include "GlobalTools.h" #include "Tools.h" #include #include namespace comm { namespace network { AwsS3Bucket getBucket(const std::string &bucketName) { return AwsS3Bucket(bucketName); } std::vector listBuckets() { Aws::S3::Model::ListBucketsOutcome outcome = getS3Client()->ListBuckets(); std::vector result; if (!outcome.IsSuccess()) { throw std::runtime_error(outcome.GetError().GetMessage()); } Aws::Vector buckets = outcome.GetResult().GetBuckets(); for (Aws::S3::Model::Bucket &bucket : buckets) { result.push_back(bucket.GetName()); } return result; } std::unique_ptr getS3Client() { Aws::Client::ClientConfiguration config; config.region = AWS_REGION; - if (tools::isDevMode()) { + if (tools::isSandbox()) { config.endpointOverride = Aws::String("localstack:4566"); config.scheme = Aws::Http::Scheme::HTTP; return std::make_unique( config, Aws::Client::AWSAuthV4Signer::PayloadSigningPolicy::Never, false); } return std::make_unique(config); } } // namespace network } // namespace comm diff --git a/services/docker-compose.yml b/services/docker-compose.yml index b5ee31d75..5c4a233e8 100644 --- a/services/docker-compose.yml +++ b/services/docker-compose.yml @@ -1,86 +1,86 @@ version: "3.9" volumes: localstack: services: # tunnelbroker tunnelbroker-server: depends_on: - localstack - rabbitmq build: dockerfile: services/tunnelbroker/Dockerfile context: ../ args: - COMM_TEST_SERVICES=${COMM_TEST_SERVICES} - - COMM_SERVICES_DEV_MODE=${COMM_SERVICES_DEV_MODE} + - COMM_SERVICES_SANDBOX=${COMM_SERVICES_SANDBOX} image: commapp/tunnelbroker-server:0.2 ports: - "${COMM_SERVICES_PORT_TUNNELBROKER}:50051" volumes: - $HOME/.aws/config:/root/.aws/config:ro - $HOME/.aws/credentials:/root/.aws/credentials:ro - ./tunnelbroker/tunnelbroker.ini:/root/tunnelbroker/tunnelbroker.ini:ro - ./tunnelbroker/tunnelbroker-dev.ini:/root/tunnelbroker/tunnelbroker-dev.ini:ro # backup backup-server: depends_on: - localstack build: dockerfile: services/backup/Dockerfile context: ../ args: - COMM_TEST_SERVICES=${COMM_TEST_SERVICES} - - COMM_SERVICES_DEV_MODE=${COMM_SERVICES_DEV_MODE} + - COMM_SERVICES_SANDBOX=${COMM_SERVICES_SANDBOX} image: commapp/backup-server:0.1 ports: - "${COMM_SERVICES_PORT_BACKUP}:50051" volumes: - $HOME/.aws/credentials:/root/.aws/credentials:ro # blob blob-server: depends_on: - localstack build: dockerfile: services/blob/Dockerfile context: ../ args: - COMM_TEST_SERVICES=${COMM_TEST_SERVICES} - - COMM_SERVICES_DEV_MODE=${COMM_SERVICES_DEV_MODE} + - COMM_SERVICES_SANDBOX=${COMM_SERVICES_SANDBOX} image: commapp/blob-server:0.1 ports: - "${COMM_SERVICES_PORT_BLOB}:50051" volumes: - $HOME/.aws/credentials:/root/.aws/credentials:ro # identity identity-server: depends_on: - localstack build: dockerfile: services/identity/Dockerfile context: ../ image: commapp/identity-server:0.1 ports: - "${COMM_SERVICES_PORT_IDENTITY}:50051" # localstack localstack: image: localstack/localstack hostname: localstack ports: - "4566:4566" environment: - SERVICES=s3,dynamodb - DATA_DIR=/tmp/localstack - HOSTNAME_EXTERNAL=localstack volumes: - localstack:/tmp/localstack # RabbitMQ rabbitmq: image: rabbitmq:3-management hostname: rabbitmq ports: - "5672:5672" - "5671:5671" - "15672:15672" environment: - RABBITMQ_DEFAULT_USER=comm - RABBITMQ_DEFAULT_PASS=comm diff --git a/services/lib/src/DynamoDBTools.cpp b/services/lib/src/DynamoDBTools.cpp index 91bc48cd1..c4a5a2883 100644 --- a/services/lib/src/DynamoDBTools.cpp +++ b/services/lib/src/DynamoDBTools.cpp @@ -1,20 +1,20 @@ #include "DynamoDBTools.h" #include "Constants.h" #include "GlobalConstants.h" #include "GlobalTools.h" namespace comm { namespace network { std::unique_ptr getDynamoDBClient() { Aws::Client::ClientConfiguration config; config.region = AWS_REGION; - if (tools::isDevMode()) { + if (tools::isSandbox()) { config.endpointOverride = Aws::String("localstack:4566"); config.scheme = Aws::Http::Scheme::HTTP; } return std::make_unique(config); } } // namespace network } // namespace comm diff --git a/services/lib/src/GlobalTools.cpp b/services/lib/src/GlobalTools.cpp index 16d3426a4..6c7cbd2e2 100644 --- a/services/lib/src/GlobalTools.cpp +++ b/services/lib/src/GlobalTools.cpp @@ -1,48 +1,48 @@ #include "GlobalTools.h" #include #include #include #include #include #include #include namespace comm { namespace network { namespace tools { uint64_t getCurrentTimestamp() { using namespace std::chrono; return duration_cast(system_clock::now().time_since_epoch()) .count(); } bool hasEnvFlag(const std::string &flag) { if (std::getenv(flag.c_str()) == nullptr) { return false; } return std::string(std::getenv(flag.c_str())) == "1"; } std::string decorateTableName(const std::string &baseName) { std::string suffix = ""; if (hasEnvFlag("COMM_TEST_SERVICES")) { suffix = "-test"; } return baseName + suffix; } -bool isDevMode() { - return hasEnvFlag("COMM_SERVICES_DEV_MODE"); +bool isSandbox() { + return hasEnvFlag("COMM_SERVICES_SANDBOX"); } std::string generateUUID() { thread_local boost::uuids::random_generator random_generator; return boost::uuids::to_string(random_generator()); } } // namespace tools } // namespace network } // namespace comm diff --git a/services/lib/src/GlobalTools.h b/services/lib/src/GlobalTools.h index 1e8a417e0..ecf6b8263 100644 --- a/services/lib/src/GlobalTools.h +++ b/services/lib/src/GlobalTools.h @@ -1,24 +1,24 @@ #pragma once #include #include namespace comm { namespace network { namespace tools { const std::string ID_SEPARATOR = ":"; uint64_t getCurrentTimestamp(); bool hasEnvFlag(const std::string &flag); std::string decorateTableName(const std::string &baseName); -bool isDevMode(); +bool isSandbox(); std::string generateUUID(); } // namespace tools } // namespace network } // namespace comm diff --git a/services/package.json b/services/package.json index 81bdbfb87..20e0b6ff1 100644 --- a/services/package.json +++ b/services/package.json @@ -1,27 +1,29 @@ { "name": "services", "version": "1.0.0", "private": true, "license": "BSD-3-Clause", "scripts": { "build-all": "./scripts/build_base_image.sh && docker-compose build", "build-base-image": "./scripts/build_base_image.sh", "run-tunnelbroker-service": "./scripts/run_server_image.sh tunnelbroker", "test-tunnelbroker-service": "./scripts/test_service.sh tunnelbroker", - "run-tunnelbroker-service-dev-mode": "COMM_SERVICES_DEV_MODE=1 ./scripts/run_server_image.sh tunnelbroker", - "test-tunnelbroker-service-dev-mode": "COMM_SERVICES_DEV_MODE=1 ./scripts/test_service.sh tunnelbroker", + "run-tunnelbroker-service-in-sandbox": "COMM_SERVICES_SANDBOX=1 ./scripts/run_server_image.sh tunnelbroker", + "test-tunnelbroker-service-in-sandbox": "COMM_SERVICES_SANDBOX=1 ./scripts/test_service.sh tunnelbroker", + "build-backup-base": "./scripts/build_base_image.sh && docker-compose build backup-base", "run-backup-service": "./scripts/run_server_image.sh backup", "test-backup-service": "./scripts/test_service.sh backup", - "run-backup-service-dev-mode": "COMM_SERVICES_DEV_MODE=1 ./scripts/run_server_image.sh backup", - "test-backup-service-dev-mode": "COMM_SERVICES_DEV_MODE=1 ./scripts/test_service.sh backup", + "run-backup-service-in-sandbox": "COMM_SERVICES_SANDBOX=1 ./scripts/run_server_image.sh backup", + "test-backup-service-in-sandbox": "COMM_SERVICES_SANDBOX=1 ./scripts/test_service.sh backup", + "build-blob-base": "./scripts/build_base_image.sh && docker-compose build blob-base", "run-blob-service": "./scripts/run_server_image.sh blob", "test-blob-service": "./scripts/test_service.sh blob", - "run-blob-service-dev-mode": "COMM_SERVICES_DEV_MODE=1 ./scripts/run_server_image.sh blob", - "test-blob-service-dev-mode": "COMM_SERVICES_DEV_MODE=1 ./scripts/test_service.sh blob", + "run-blob-service-in-sandbox": "COMM_SERVICES_SANDBOX=1 ./scripts/run_server_image.sh blob", + "test-blob-service-in-sandbox": "COMM_SERVICES_SANDBOX=1 ./scripts/test_service.sh blob", "run-all-services": "./scripts/run_all_services.sh", "test-all-services": "./scripts/test_all_services.sh", - "test-all-services-dev-mode": "COMM_SERVICES_DEV_MODE=1 ./scripts/test_all_services.sh", - "run-all-services-dev-mode": "COMM_SERVICES_DEV_MODE=1 ./scripts/run_all_services.sh", + "test-all-services-in-sandbox": "COMM_SERVICES_SANDBOX=1 ./scripts/test_all_services.sh", + "run-all-services-in-sandbox": "COMM_SERVICES_SANDBOX=1 ./scripts/run_all_services.sh", "init-local-cloud": "./scripts/init_local_cloud.sh" } -} +} \ No newline at end of file diff --git a/services/tunnelbroker/Dockerfile b/services/tunnelbroker/Dockerfile index b566993ab..abf3cd115 100644 --- a/services/tunnelbroker/Dockerfile +++ b/services/tunnelbroker/Dockerfile @@ -1,35 +1,35 @@ FROM commapp/services-base:1.1 ARG MAKE_JOBS=4 ENV MAKEFLAGS="-j${MAKE_JOBS}" WORKDIR /transferred/scripts # Install SDKs COPY services/tunnelbroker/docker/install_amqp_cpp.sh . RUN ./install_amqp_cpp.sh COPY services/tunnelbroker/docker/install_cryptopp.sh . RUN ./install_cryptopp.sh COPY services/tunnelbroker/docker/install_libuv.sh . RUN ./install_libuv.sh ARG COMM_TEST_SERVICES -ARG COMM_SERVICES_DEV_MODE +ARG COMM_SERVICES_SANDBOX ENV COMM_TEST_SERVICES=${COMM_TEST_SERVICES} -ENV COMM_SERVICES_DEV_MODE=${COMM_SERVICES_DEV_MODE} +ENV COMM_SERVICES_SANDBOX=${COMM_SERVICES_SANDBOX} WORKDIR /transferred COPY native/cpp/CommonCpp/grpc/protos/tunnelbroker.proto protos/tunnelbroker.proto COPY services/lib/cmake-components cmake-components COPY services/lib/docker/ scripts/ COPY services/tunnelbroker/docker/* docker/ COPY services/tunnelbroker/ . COPY services/lib/src/* src/ RUN scripts/build_service.sh CMD if [ "$COMM_TEST_SERVICES" -eq 1 ]; then scripts/run_service.sh; else scripts/run_service.sh; fi diff --git a/services/tunnelbroker/src/server.cpp b/services/tunnelbroker/src/server.cpp index 8a460bb7a..984b5d52e 100644 --- a/services/tunnelbroker/src/server.cpp +++ b/services/tunnelbroker/src/server.cpp @@ -1,67 +1,67 @@ #include "AmqpManager.h" #include "ConfigManager.h" #include "Constants.h" #include "GlobalTools.h" #include "TunnelbrokerServiceImpl.h" #include #include #include #include #include namespace comm { namespace network { void RunServer() { TunnelBrokerServiceImpl service; grpc::EnableDefaultHealthCheckService(true); grpc::ServerBuilder builder; // Listen on the given address without any authentication mechanism. builder.AddListeningPort( SERVER_LISTEN_ADDRESS, grpc::InsecureServerCredentials()); // Register "service" as the instance through which we'll communicate with // clients. In this case it corresponds to an *synchronous* service. builder.RegisterService(&service); std::unique_ptr server(builder.BuildAndStart()); LOG(INFO) << "gRPC Server listening at :" << SERVER_LISTEN_ADDRESS; // Wait for the server to shutdown. Note that some other thread must be // responsible for shutting down the server for this call to ever return. server->Wait(); } void RunAmqpClient() { AmqpManager::getInstance().connect(); } void InitLogging(const char *programName) { FLAGS_logtostderr = true; FLAGS_colorlogtostderr = true; - if (comm::network::tools::isDevMode()) { + if (comm::network::tools::isSandbox()) { // Log levels INFO, WARNING, ERROR, FATAL are 0, 1, 2, 3, respectively FLAGS_minloglevel = 0; } else { FLAGS_minloglevel = 1; } google::InitGoogleLogging(programName); } } // namespace network } // namespace comm int main(int argc, char **argv) { comm::network::InitLogging(argv[0]); - if (comm::network::tools::isDevMode()) { + if (comm::network::tools::isSandbox()) { comm::network::config::ConfigManager::getInstance().load( comm::network::DEV_CONFIG_FILE_PATH); } else { comm::network::config::ConfigManager::getInstance().load( comm::network::CONFIG_FILE_PATH); } std::thread amqpThread(comm::network::RunAmqpClient); std::thread grpcThread(comm::network::RunServer); amqpThread.join(); grpcThread.join(); return 0; }