diff --git a/docs/dev_services.md b/docs/dev_services.md index 3e8f7b3d4..0c990257e 100644 --- a/docs/dev_services.md +++ b/docs/dev_services.md @@ -1,113 +1,113 @@ # Requirements At the moment, our services can be built and run on Linux and macOS via Docker. Unfortunately, Windows is not supported at this time. You’ll ideally want a machine with at least 16 GiB of RAM because running a Docker container can consume up to 4 GiB of RAM. We use Ubuntu as the base Docker image for services. # Prerequisites ## Docker To build and run the services you need to install [Docker](https://docs.docker.com/desktop/) and [Docker Compose](https://docs.docker.com/compose/install) on your system. ## Node We use the `yarn` package manager to install dependencies and run scripts. Installation instructions can be found in the [dev_environment doc](https://github.com/CommE2E/comm/blob/master/docs/dev_environment.md#node). ## AWS Some of our services access AWS resources via the AWS C++ SDK. To access these resources, you'll need to configure the `~/.aws/credentials` and `~/.aws/config` files correctly on your host machine. Instructions for setting these configuration files can be found [here](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html). We recommend running `aws configure`, which will prompt you for the necessary configuration values. ## Terraform We use [Terraform](https://www.terraform.io/) to create and manage our AWS resources. Installation instructions can be found [here](https://www.terraform.io/downloads). ## RabbitMQ (Tunnelbroker only) [RabbitMQ](https://www.rabbitmq.com/) is an open-source message broker service. We use RabbitMQ in Tunnelbroker to facilitate communication between devices and keyservers. We use the secure AMQPS protocol to connect to RabbitMQ instances hosted on AWS. In order to access and manage RabbitMQ instances, you'll need credentials and the [proper permissions](https://www.rabbitmq.com/access-control.html). You can add new users or edit permissions for existing ones through the [RabbitMQ Management plugin](https://www.rabbitmq.com/management.html). Alternatively, you can manage credentials and permissions from the `rabbitmqctl` CLI. For example, to add a new user you can run the following command: ``` rabbitmqctl add_user {username} ``` You'll need to create a Tunnelbroker-specific configuration file. ``` -mkdir -p $HOME/tunnelbroker -vim $HOME/tunnelbroker/tunnelbroker.ini +mkdir -p $HOME/.config +vim $HOME/.config/tunnelbroker.ini ``` -Provide a unique ID for each running instance of Tunnelbroker and a RabbitMQ URI in accordance with this [specification](https://www.rabbitmq.com/uri-spec.html). +Provide a unique ID for each running instance of Tunnelbroker and a RabbitMQ URI in accordance with this [specification](https://www.rabbitmq.com/uri-spec.html): ``` [tunnelbroker] instance-id = tunnelbroker1 [keyserver] default_keyserver_id = ks:256 [amqp] uri = amqp://guest:guest@0.0.0.0/vhost ``` # Building and running `services/package.json` provides useful scripts to build and run services. The `run` scripts will automatically build the services if necessary and run them. You can find the full list of scripts [here](https://github.com/CommE2E/comm/blob/master/services/package.json) in the `scripts` section. # Developing and debugging ## Visual Studio Code If you are using Visual Studio Code as your code editor you can [attach to a Docker container](https://code.visualstudio.com/docs/remote/attach-container) and develop inside it. ## Sandbox environment for services You can run the Comm services locally in a sandbox environment for development and testing purposes. The sandbox uses a [local cloud stack](https://localstack.cloud/) that includes DynamoDB and S3 running locally in Docker containers. The sandbox also includes a [RabbitMQ](https://www.rabbitmq.com/) Docker container, which is required by Tunnelbroker. ### Configuration changes in the sandbox -In your sandbox, services will connect to a local cloud stack, ignoring the `~/.aws` connection settings. The `-test` suffix is applied for all DynamoDB table names in this mode. Tunnelbroker will use the `services/tunnelbroker/tunnelbroker-dev.ini` configuration file and connect to a local instance of the RabbitMQ server. +In your sandbox, services will connect to a local cloud stack, ignoring the `~/.aws` connection settings. The `-test` suffix is applied for all DynamoDB table names in this mode. The log level in this mode is increased from ERROR to INFO. ### Running services in the sandbox First, you need to initialize the local cloud using the following command from the the `services` directory: ``` yarn init-local-cloud ``` This will start the LocalStack Docker image and initialize required resources, including DynamoDB tables and S3 buckets, using the Terraform scripts located in `services/terraform`. To start a certain service in the sandbox you can run the following command: ``` yarn run-[service-name]-service-in-sandbox ``` For example, for Tunnelbroker the command will look like this: ``` yarn run-tunnelbroker-service-in-sandbox ``` You can also run all services at once in the sandbox using the command below: ``` yarn run-all-services-in-sandbox ``` ### Rebuilding the base image If you ever wish to rebuild the base image, you should get a tool named [buildx](https://github.com/docker/buildx). It should be attached with the Docker desktop app on the macOS, but if you use Linux, you will probably need to install it manually. For the installation instructions, please go [here](https://github.com/docker/buildx#installing). diff --git a/services/tunnelbroker/src/libcpp/src/Constants.h b/services/tunnelbroker/src/libcpp/src/Constants.h index 6f53c2f30..b017627a8 100644 --- a/services/tunnelbroker/src/libcpp/src/Constants.h +++ b/services/tunnelbroker/src/libcpp/src/Constants.h @@ -1,72 +1,71 @@ #pragma once #include #include #include namespace comm { namespace network { // AWS DynamoDB const size_t DYNAMODB_MAX_BATCH_ITEMS = 25; const size_t DYNAMODB_BACKOFF_FIRST_RETRY_DELAY = 50; const size_t DYNAMODB_MAX_BACKOFF_TIME = 10000; // 10 seconds const std::string DEVICE_SESSIONS_TABLE_NAME = "tunnelbroker-device-sessions"; const std::string DEVICE_SESSIONS_VERIFICATION_MESSAGES_TABLE_NAME = "tunnelbroker-verification-messages"; const std::string DEVICE_PUBLIC_KEY_TABLE_NAME = "tunnelbroker-public-keys"; const std::string MESSAGES_TABLE_NAME = "tunnelbroker-messages"; // Sessions const size_t SIGNATURE_REQUEST_LENGTH = 64; const size_t SESSION_ID_LENGTH = 64; const size_t SESSION_RECORD_TTL = 30 * 24 * 3600; // 30 days const size_t SESSION_SIGN_RECORD_TTL = 24 * 3600; // 24 hours const std::regex SESSION_ID_FORMAT_REGEX( "[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}"); // AMQP (RabbitMQ) const std::string AMQP_FANOUT_EXCHANGE_NAME = "allBrokers"; // Message broker queue message TTL const size_t AMQP_MESSAGE_TTL = 300 * 1000; // 5 min // queue TTL in case of no consumers (tunnelbroker is down) const size_t AMQP_QUEUE_TTL = 24 * 3600 * 1000; // 24 hours // routing message headers name const std::string AMQP_HEADER_FROM_DEVICEID = "fromDeviceID"; const std::string AMQP_HEADER_TO_DEVICEID = "toDeviceID"; const std::string AMQP_HEADER_MESSAGEID = "messageID"; const size_t AMQP_RECONNECT_ATTEMPT_INTERVAL_MS = 3000; const size_t AMQP_RECONNECT_MAX_ATTEMPTS = 10; // DeviceID // DEVICEID_CHAR_LENGTH has to be kept in sync with deviceIDCharLength // which is defined in web/utils/device-id.js // and with DEVICE_ID_CHAR_LENGTH // defined in native/native_rust_library/src/crypto_tools.rs const size_t DEVICEID_CHAR_LENGTH = 64; // DEVICEID_FORMAT_REGEX has to be kept in sync with deviceIDFormatRegex // which is defined in web/utils/device-id.js // and with DEVICE_ID_FORMAT_REGEX // defined in native/native_rust_library/src/crypto_tools.rs const std::regex DEVICEID_FORMAT_REGEX( "^(ks|mobile|web):[a-zA-Z0-9]{" + std::to_string(DEVICEID_CHAR_LENGTH) + "}$"); const size_t DEVICE_ONLINE_PING_INTERVAL_MS = 3000; // Config const std::string CONFIG_FILE_DIRECTORY_ENV_VARIABLE = "TUNNELBROKER_CONFIG_FILE_DIRECTORY"; const std::string DEFAULT_CONFIG_FILE_DIRECTORY = - std::string(std::getenv("HOME")) + "/tunnelbroker"; + std::string(std::getenv("HOME")) + "/.config"; const std::string CONFIG_FILE_NAME = "tunnelbroker.ini"; -const std::string SANDBOX_CONFIG_FILE_NAME = "tunnelbroker-sandbox.ini"; // DeliveryBroker const size_t DELIVERY_BROKER_MAX_QUEUE_SIZE = 100; // Database messages TTL const size_t MESSAGE_RECORD_TTL = 300 * 24 * 60 * 60; // 300 days } // namespace network } // namespace comm diff --git a/services/tunnelbroker/src/libcpp/src/Tools/ConfigManager.cpp b/services/tunnelbroker/src/libcpp/src/Tools/ConfigManager.cpp index 5c5502de2..ba0313ae1 100644 --- a/services/tunnelbroker/src/libcpp/src/Tools/ConfigManager.cpp +++ b/services/tunnelbroker/src/libcpp/src/Tools/ConfigManager.cpp @@ -1,150 +1,146 @@ #include "ConfigManager.h" #include "Constants.h" #include "GlobalTools.h" #include namespace comm { namespace network { namespace config { const std::string ConfigManager::OPTION_TUNNELBROKER_ID = "tunnelbroker.instance-id"; const std::string ConfigManager::OPTION_DEFAULT_KEYSERVER_ID = "keyserver.default_keyserver_id"; const std::string ConfigManager::OPTION_AMQP_URI = "amqp.uri"; const std::string ConfigManager::OPTION_AMQP_FANOUT_EXCHANGE = "amqp.fanout_exchange_name"; const std::string ConfigManager::OPTION_DYNAMODB_SESSIONS_TABLE = "dynamodb.sessions_table_name"; const std::string ConfigManager::OPTION_DYNAMODB_SESSIONS_VERIFICATION_TABLE = "dynamodb.sessions_verification_table_name"; const std::string ConfigManager::OPTION_DYNAMODB_SESSIONS_PUBLIC_KEY_TABLE = "dynamodb.sessions_public_key_table_name"; const std::string ConfigManager::OPTION_DYNAMODB_MESSAGES_TABLE = "dynamodb.messages_table_name"; const std::string ConfigManager::OPTION_NOTIFS_APNS_P12_CERT_PATH = "notifications.apns_cert_path"; const std::string ConfigManager::OPTION_NOTIFS_APNS_P12_CERT_PASSWORD = "notifications.apns_cert_password"; const std::string ConfigManager::OPTION_NOTIFS_APNS_TOPIC = "notifications.apns_topic"; const std::string ConfigManager::OPTION_NOTIFS_FCM_SERVER_KEY = "notifications.fcm_server_key"; ConfigManager &ConfigManager::getInstance() { static ConfigManager instance; return instance; } void ConfigManager::load() { char const *configFileDirectoryFromEnvironment = std::getenv(CONFIG_FILE_DIRECTORY_ENV_VARIABLE.c_str()); std::string configFilePath = DEFAULT_CONFIG_FILE_DIRECTORY; if (configFileDirectoryFromEnvironment != nullptr) { configFilePath = std::string{configFileDirectoryFromEnvironment}; } - if (comm::network::tools::isSandbox()) { - loadConfigFile(configFilePath + "/" + SANDBOX_CONFIG_FILE_NAME); - } else { - loadConfigFile(configFilePath + "/" + CONFIG_FILE_NAME); - } + loadConfigFile(configFilePath + "/" + CONFIG_FILE_NAME); } void ConfigManager::loadConfigFile(const std::string configFilePath) { try { std::ifstream fileStream; fileStream.open(configFilePath.c_str(), std::ifstream::in); if (!fileStream.is_open()) { throw std::runtime_error("Error: can not open file " + configFilePath); } boost::program_options::options_description description{ "Tunnelbroker options"}; description.add_options()( this->OPTION_TUNNELBROKER_ID.c_str(), boost::program_options::value()->required(), "Tunnelbroker unique identification"); description.add_options()( this->OPTION_DEFAULT_KEYSERVER_ID.c_str(), boost::program_options::value()->required(), "Default and only allowed keyserver deviceID"); description.add_options()( this->OPTION_AMQP_URI.c_str(), boost::program_options::value()->required(), "AMQP URI connection string"); description.add_options()( this->OPTION_AMQP_FANOUT_EXCHANGE.c_str(), boost::program_options::value()->default_value( AMQP_FANOUT_EXCHANGE_NAME), "AMQP Fanout exchange name"); description.add_options()( this->OPTION_DYNAMODB_SESSIONS_TABLE.c_str(), boost::program_options::value()->default_value( DEVICE_SESSIONS_TABLE_NAME), "DynamoDB table name for sessions"); description.add_options()( this->OPTION_DYNAMODB_SESSIONS_VERIFICATION_TABLE.c_str(), boost::program_options::value()->default_value( DEVICE_SESSIONS_VERIFICATION_MESSAGES_TABLE_NAME), "DynamoDB table name for sessions verification messages"); description.add_options()( this->OPTION_DYNAMODB_SESSIONS_PUBLIC_KEY_TABLE.c_str(), boost::program_options::value()->default_value( DEVICE_PUBLIC_KEY_TABLE_NAME), "DynamoDB table name for public keys"); description.add_options()( this->OPTION_DYNAMODB_MESSAGES_TABLE.c_str(), boost::program_options::value()->default_value( MESSAGES_TABLE_NAME), "DynamoDB table name for messages"); description.add_options()( this->OPTION_NOTIFS_APNS_P12_CERT_PATH.c_str(), boost::program_options::value()->required(), "P12 certificate path for iOS notifications"); description.add_options()( this->OPTION_NOTIFS_APNS_P12_CERT_PASSWORD.c_str(), boost::program_options::value()->required(), "P12 certificate password for iOS notifications"); description.add_options()( this->OPTION_NOTIFS_APNS_TOPIC.c_str(), boost::program_options::value()->required(), "APNs messages topic for iOS notifications"); description.add_options()( this->OPTION_NOTIFS_FCM_SERVER_KEY.c_str(), boost::program_options::value()->required(), "Firebase Cloud Messaging server key for Android notifications"); boost::program_options::parsed_options parsedDescription = boost::program_options::parse_config_file( fileStream, description, true); boost::program_options::store(parsedDescription, this->variablesMap); boost::program_options::notify(this->variablesMap); fileStream.close(); } catch (const std::exception &e) { throw std::runtime_error( "Got an exception at ConfigManager: " + std::string(e.what())); } } std::string ConfigManager::getParameter(std::string param) { if (!this->variablesMap.count(param) && !this->variablesMap[param].defaulted()) { throw std::runtime_error( "ConfigManager Error: config parameter " + param + " is not set."); } const std::string parameterValue = this->variablesMap[param].as(); if (parameterValue.empty()) { throw std::runtime_error( "ConfigManager Error: config parameter " + param + " can not be empty."); } return parameterValue; } } // namespace config } // namespace network } // namespace comm diff --git a/services/tunnelbroker/tunnelbroker-sandbox.ini b/services/tunnelbroker/tunnelbroker.ini similarity index 100% rename from services/tunnelbroker/tunnelbroker-sandbox.ini rename to services/tunnelbroker/tunnelbroker.ini