It’s primarily because Apple only supports iOS development using macOS. It’s true that we could support web, keyserver, and Android development on other operating systems, but because of the Apple requirement, all of our active developers currently run macOS. We’d very much welcome a PR to build out support on Windows!
Unfortunately the dev environment is overall pretty heavy. You’ll ideally want a machine with at least 32 GiB of RAM, although 16 GiB should suffice.
# Prerequisites
## Xcode
Go to the [Mac App Store](https://apps.apple.com/us/app/xcode/id497799835) to install Xcode, or if you already have it, to update it to the latest version.
Once Xcode is installed, open it up. If you are prompted, follow the instructions to install any “Additional Required Components”.
Finally, you need to make sure that the Xcode “Command Line Tools” are installed. Run this command:
```
xcode-select --install
```
## Homebrew
Install [Homebrew](https://brew.sh/), a package manager for macOS.
## Node
Next, install [Node](https://nodejs.org/) using Homebrew. We’re going to use version 16 to avoid some possible issues that come up on Apple silicon when we install project dependencies.
```
brew install node@16 && brew upgrade node@16
```
The reason we use both `install` and `upgrade` is that there’s no single Homebrew command equivalent to “install if not installed, and upgrade if already installed”.
## PHP
[PHP](https://www.php.net) is needed for Arcanist. As of macOS 12 (Monterey), PHP is no longer bundled with the OS and needs to be installed via Homebrew.
```
brew install php@7.4 && brew upgrade php@7.4
```
## Rust
We use a Rust [implementation](https://github.com/novifinancial/opaque-ke) of the OPAQUE password-authenticated key exchange protocol, so you will need to install Rust to compile the static library. The easiest way to do this is with `rustup`.
```
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
```
## ShellCheck
[ShellCheck](https://www.shellcheck.net/) is a static analysis tool that provides warnings and suggestions for shell scripts. We’ll install ShellCheck using Homebrew.
```
brew install shellcheck && brew upgrade shellcheck
```
## Yarn
We use the [Yarn](https://yarnpkg.com/) package manager for JavaScript in our repo.
```
brew install yarn && brew upgrade yarn
```
## Watchman
Watchman is a tool from Facebook used in the React Native dev environment to watch for changes to your filesystem.
```
brew install watchman && brew upgrade watchman
```
## Node Version Manager
Node Version Manager (nvm) is a tool that ensures we use the same version of Node on our keyserver between prod and dev environments.
```
brew install nvm && brew upgrade nvm
```
After installing, Homebrew will print out some instructions under the Caveats section of its output. It will ask you to do two things: `mkdir ~/.nvm`, and to add some lines to your `~/.bash_profile` (or desired shell configuration file). We recommend that you append `--no-use` to the line that loads nvm, so that you continue to use your Homebrew-sourced Node distribution by default.
These lines are different depending on if you’re on Apple silicon or Intel x86-64. If you’re on Apple silicon, you should add the following to your shell configuration file (note the addition of the `--no-use` flag on the line that loads nvm):
```
export NVM_DIR="$HOME/.nvm"
[ -s "/opt/homebrew/opt/nvm/nvm.sh" ] && \. "/opt/homebrew/opt/nvm/nvm.sh" --no-use # This loads nvm
[ -s "/opt/homebrew/opt/nvm/etc/bash_completion.d/nvm" ] && \. "/opt/homebrew/opt/nvm/etc/bash_completion.d/nvm" # This loads nvm bash_completion
```
Otherwise, if you’re on Intel x86-64, add the following to your shell configuration file:
```
export NVM_DIR="$HOME/.nvm"
[ -s "/usr/local/opt/nvm/nvm.sh" ] && \. "/usr/local/opt/nvm/nvm.sh" --no-use # This loads nvm
[ -s "/usr/local/opt/nvm/etc/bash_completion.d/nvm" ] && \. "/usr/local/opt/nvm/etc/bash_completion.d/nvm" # This loads nvm bash_completion
```
Now either close and reopen your terminal window or re-source your shell configuration file in order to load nvm:
```
source ~/.bash_profile
```
## MariaDB
For the keyserver database we use MariaDB, which is a community-driven fork of MySQL. In this step we’ll install MariaDB using Homebrew.
```
brew install mariadb && brew upgrade mariadb
```
Next we’ll configure MariaDB to start when your computer boots using `brew services`:
```
brew tap homebrew/services
brew services start mariadb
```
Finally, you should set up a root password for your local MariaDB instance. To do this we’ll use the `mysqladmin` command. Note that many of MariaDB’s commands still use the “mysql” name for compatibility.
```
sudo mysqladmin -u root password
```
## Redis
We use Redis on the keyserver side as a message broker.
```
brew install redis && brew upgrade redis
```
We’ll set it up to start on boot with `brew services`:
```
brew services start redis
```
## CocoaPods
CocoaPods is a dependency management system for iOS development. React Native uses it to manage native modules.
```
brew install cocoapods && brew upgrade cocoapods
```
## Reactotron
Reactotron is an event tracker and logger that can be used to aid in debugging on React Native.
```
brew install reactotron && brew upgrade reactotron
```
## React Dev Tools Chrome extension
The React Dev Tools Chrome extension lets you inspect the React component tree for web applications in Chrome. You can install it by navigating [here](https://chrome.google.com/webstore/detail/react-developer-tools/fmkadmapgofadopljbjfkapdkoienihi) on Chrome.
## Redux Dev Tools Chrome extension
The Redux Dev Tools Chrome extension lets you watch for Redux actions and inspect the Redux store state, both for web applications in Chrome, but also for our native applications using the “Remote DevTools” functionality. To install it, navigate [here](https://chrome.google.com/webstore/detail/redux-devtools/lmhkpmbekcpmknklioeibfkpmmfibljd) on Chrome.
## JDK
We’ll need the Java Development Kit (JDK) for Android development. We’re using [SDKMAN!](https://sdkman.io/) to manage our JDK installation.
Run the following to install SDKMAN!:
```
curl -s "https://get.sdkman.io" | bash
```
Now either close and reopen your terminal window or re-source your `~/.bash_profile` (or desired shell configuration file) in order to load SDKMAN!:
```
source ~/.bash_profile
```
You can run `sdk version` to see if SDKMAN! was installed properly.
Run the following to install Azul Zulu 11 with SDKMAN!:
```
sdk install java 11.0.13-zulu
```
SDKMAN! takes care of setting up the `$JAVA_HOME` environment variable to point to the newly installed JDK. You can verify this by running `echo $JAVA_HOME`.
## Android Studio
Start by downloading and installing [Android Studio](https://developer.android.com/studio/index.html) for your platform. When prompted to choose an installation type, select “Custom”.
You’ll be prompted to select a JDK installation. If your SDKMAN!-sourced JDK doesn’t appear in the dropdown, you can find the absolute path to your installed JDK with the following command:
```
sdk home java 11.0.13-zulu
```
Make sure you check the boxes for the following:
#### Intel x86-64:
- `Android SDK`
- `Android SDK Platform`
- `Performance (Intel ® HAXM)`
- `Android Virtual Device`
#### Apple silicon:
- `Android SDK`
- `Android SDK Platform`
- `Android Virtual Device`
### Android SDK
Android Studio installs the latest Android SDK by default, but since React Native uses the Android 11 SDK specifically, we’ll need to install it using Android Studio’s SDK Manager. You can access the SDK Manager from the “Welcome to Android Studio” screen that pops up when you first open the application, from More Actions → SDK Manager. If you already have a project open, you can access it from Tools → SDK Manager.
Once you have the SDK Manager open, select the “SDK Platforms” tab, and then check the box for “Show Package Details”. Now expand the “Android 11 (R)” section, and make sure the following subsections are checked:
- `Android SDK Platform 30`
#### Intel x86-64:
- `Intel x86 Atom_64 System Image` or `Google APIs Intel x86 Atom System Image`
#### Apple silicon:
- `Google Play ARM 64 v8a System Image`
Next, select the “SDK Tools” tab, and check the box for “Show Package Details”. Refer to `native/android/build.gradle` for specific tool versions and install the following:
- Android SDK Build-Tools
- NDK
- CMake version 3.18.1
To finish the SDK Manager step, click “Apply” to download and install everything you’ve selected.
### Enable Android CLI commands
You’ll need to append the following lines to your `~/.bash_profile` (or desired shell configuration file) in order for React Native to be able to build your Android project.
```
export ANDROID_HOME=$HOME/Library/Android/sdk
export PATH=$PATH:$ANDROID_HOME/emulator
export PATH=$PATH:$ANDROID_HOME/tools
export PATH=$PATH:$ANDROID_HOME/tools/bin
export PATH=$PATH:$ANDROID_HOME/platform-tools
```
Now either close and reopen your terminal window or re-source your shell configuration file in order to run the new commands:
```
source ~/.bash_profile
```
## Arcanist
We use Phabricator for code review. To upload a “diff” to Phabricator, you’ll need to use a tool called Arcanist.
To install Arcanist, we’ll need to clone its Git repository. Pick a place in your filesystem to store it, and then run this command:
```
git clone https://github.com/phacility/arcanist.git
```
Next, you’ll need to add the path `./arcanist/bin` to your `$PATH` in your `~/.bash_profile` (or desired shell configuration file):
```
export PATH=$PATH:~/src/arcanist/bin
```
Make sure to replace the `~/src` portion of the above with the location of the directory you installed Arcanist in.
# Configuration
## MariaDB
Next we’ll set up a MariaDB user and a fresh database. We’ll start by opening up a console using the `mysql` command.
```
mysql -u root -p
```
Type in the MariaDB root password you set up previously when prompted. Then, we’ll go ahead and create an empty database.
```
CREATE DATABASE comm;
```
Now we need to create a user that can access this database. For the following command, replace “password” with a unique password.
```
CREATE USER comm@localhost IDENTIFIED BY 'password';
```
Finally, we will give permissions to this user to access this database.
```
GRANT ALL ON comm.* TO comm@localhost;
```
You can now exit the MariaDB console using Ctrl+D.
## TablePlus
Feel free to use any MariaDB administration platform that you’re comfortable with. PHP was deprecated in macOS 12 (Monterey), leading many of us to switch to [TablePlus](https://tableplus.com/).
After installing TablePlus, you need to open a new connection. After opening TablePlus, click the “Create a new connection” text at the bottom of the window that appears.
- Alternatively, you can navigate through Connection → New... in the menu at the top of the display.
Choose MariaDB from the database options that appear. You’ll be prompted for:
- Name (Comm)
- Host (localhost)
- Port (3306 by default)
- User (comm)
- Password (the one you made when initializing the MariaDB server in the previous step)
## Android Emulator
In order to test the Android app on your computer you’ll need to set up an Android Emulator. To do this we’ll need to open up the Virtual Device Manager in Android Studio. You can access the Virtual Device Manager from the “Welcome to Android Studio” screen that pops up when you first open the application, from More Actions → Virtual Device Manager. If you already have a project open, you can access it from Tools → Virtual Device Manager.
With the Virtual Device Manager open, select “Create device” on the top left. Feel free to select any “device definition” that includes Play Store support.
On the next screen you’ll be asked to select a system image. Go for the latest version of Android that’s been released.
From there you can just hit Next and then Finish. You should then be able to start your new Android Virtual Device from the Virtual Device Manager.
# Git repo
## Clone from GitHub
Finally! It’s time to clone the repo from GitHub.
```
git clone git@github.com:CommE2E/comm.git
```
Once you have the repo cloned, run this command to pull in dependencies.
```
cd comm
yarn cleaninstall
```
## URLs
The keyserver needs to know some info about paths in order to properly construct URLs.
```
mkdir -p keyserver/facts
-vim keyserver/facts/commapp_url.json
+vim keyserver/facts/webapp_url.json
```
-Your `commapp_url.json` file should look like this:
+Your `webapp_url.json` file should look like this:
```json
{
"baseDomain": "http://localhost:3000",
- "basePath": "/comm/",
- "baseRoutePath": "/comm/",
+ "basePath": "/webapp/",
+ "baseRoutePath": "/webapp/",
"https": false,
"proxy": "none"
}
```
Additionally, we’ll create a file for the URLs in the landing page.
```
vim keyserver/facts/landing_url.json
```
Your `landing_url.json` file should look like this:
```json
{
"baseDomain": "http://localhost:3000",
"basePath": "/commlanding/",
"baseRoutePath": "/commlanding/",
"https": false,
"proxy": "none"
}
```
## MariaDB
The keyserver side needs to see some config files before things can work. The first is a config file with MariaDB details.
```
cd keyserver
mkdir secrets
vim secrets/db_config.json
```
The DB config file should look like this:
```json
{
"host": "127.0.0.1",
"user": "comm",
"password": "password",
"database": "comm",
"dbType": "mariadb10.8"
}
```
Make sure to replace the password with the one you set up for your `comm` MariaDB user earlier.
## Olm
The second config file contains some details that the keyserver needs in order to launch Olm sessions to provide E2E encryption.
We have a script that can generate this file for you. However, before we can run the script we’ll need to use Babel to transpile our source files into something Node can interpret. Babel will transpile the files in `src` into a new directory called `dist`. We’ll also use `rsync` to copy over files that don’t need transpilation.
```
cd keyserver
yarn babel-build
yarn rsync
yarn script dist/scripts/generate-olm-config.js
```
This script will create the `keyserver/secrets/olm_config.json` config file.
## Phabricator
The last configuration step is to set up an account on Phabricator, where we handle code review. Start by [logging in to Phabricator](https://phab.comm.dev) using your GitHub account.
Next, make sure you’re inside the directory containing the Comm Git repository, and run the following command:
```
arc install-certificate
```
This command will help you connect your Phabricator account with the local Arcanist instance, allowing you to run `arc diff` and `arc land` commands.
# Development
## Flow typechecker
It’s good to run the `flow` typechecker frequently to make sure you’re not introducing any type errors. Flow treats each Yarn Workspace as a separate environment, and as such runs a separate type-checking server for each. This server is started when you first run `node_modules/.bin/flow` in each of the four Yarn Workspace folders.
To make sure Flow runs from the command-line, you can edit your `$PATH` environmental variable in your `~/.bash_profile` file (or desired shell configuration file) to always include `./node_modules/.bin`.
```
export PATH=$PATH:./node_modules/.bin
```
As always, make sure you reload the `~/.bash_profile` after editing it:
```
source ~/.bash_profile
```
You should now be able to run `flow` in any of the Yarn workspaces:
```
cd lib
flow
```
## Running keyserver
To run the web app, landing page, or the mobile app on the iOS Simulator or Android Emulator, you’ll need to run the keyserver.
Open a new terminal and run:
```
cd keyserver
yarn dev
```
This command runs three processes. The first two are to keep the `dist` folder updated whenever the `src` folder changes. They are “watch” versions of the same Babel and `rsync` commands we used to initially create the `dist` folder (before running the `generate-olm-config.js` script above). The final process is `nodemon`, which is similar to `node` except that it restarts whenever any of its source files (in the `dist` directory) changes.
Note that if you run `yarn dev` in `keyserver` right after `yarn cleaninstall`, before Webpack is given a chance to build `app.build.cjs`/`landing.build.cjs` files, then Node will crash when it attempts to import those files. Just make sure to run `yarn dev` (or `yarn prod`) in `web` or `landing` before attempting to load the corresponding webpages.
## Running web app
First, make sure that the keyserver is running. If you haven’t already, run:
```
cd keyserver
yarn dev
```
Next, open a new terminal and run:
```
cd web
yarn dev
```
-You should now be able to load the web app in your web browser at http://localhost/comm/.
+You should now be able to load the web app in your web browser at http://localhost/webapp/.
This command will start two processes. One is `webpack-dev-server`, which will serve the JS files. `webpack-dev-server` also makes sure the website automatically hot-reloads whenever any of the source files change. The other process is `webpack --watch`, which will build the `app.build.cjs` file, as well as rebuilding it whenever any of the source files change. The `app.build.cjs` file is consumed by the Node server in order to pre-render the initial HTML from the web source (“Server-Side Rendering”).
## Running desktop app
First, make sure that the keyserver and the web app is running. If you haven’t already, run:
```
cd keyserver
yarn dev
```
Next, open a new terminal and run:
```
cd web
yarn dev
```
Then start the desktop app:
```
cd desktop
yarn dev
```
This will run the desktop app in dev mode. Only code that is shared with the web app will be hot-reloaded, but you can easily reload the Electron app by typing `rs` into the terminal.
## Running landing page
First, make sure that the keyserver is running. If you haven’t already, run:
```
cd keyserver
yarn dev
```
Next, open a new terminal and run:
```
cd landing
yarn dev
```
You should now be able to load the landing page in your web browser at http://localhost/commlanding/.
This runs the same two processes as the web app, but for the landing page. Note that the `landing.build.cjs` file (similar to the web app’s `app.build.cjs` file) is consumed by the Node server.
## Running mobile app on iOS Simulator
First, make sure that the keyserver is running. If you haven’t already, run:
```
cd keyserver
yarn dev
```
Next, make sure that the Metro bundler is running. If you haven’t already, open a new terminal and run:
```
cd native
yarn dev
```
This command runs the Metro bundler, which handles bundling our app’s JavaScript code and communicating with the debug build of the app running on either a physical or virtual device.
Finally, open `native/ios/Comm.xcworkspace` in Xcode. Select a Simulator from the Scheme menu in the Workspace Toolbar. Then hit the Run button to build and run the project.
## Running mobile app on Android Emulator
First, make sure that the keyserver is running. If you haven’t already, run:
```
cd keyserver
yarn dev
```
Next, make sure that the Metro bundler is running. If you haven’t already, open a new terminal and run:
```
cd native
yarn dev
```
Next, boot up an Android Emulator using Android Studio’s Virtual Device Manager. You should have a single Android Emulator (or plugged-in device) running at one time.
Finally, use this command to build and run the Android app:
```
cd native
yarn react-native run-android
```
## Running mobile app on physical iOS devices
There are a few things you’ll need to do before you can deploy the app to a physical iOS device.
### Xcode settings
First, in Xcode, open the Comm workspace `native/ios/Comm.xcworkspace`. Make sure that you’re signed into Xcode with an Apple Developer account (either the Comm developer team’s or your own). You can see any accounts currently associated with Xcode by navigating to Xcode → Settings → Accounts.
Next, you’ll want to ensure that the Comm project is configured with a valid Team, Bundle Identifier, and Provisioning Profile. To access these settings, navigate to View → Navigators → Project, and select the “Comm” project in the left sidebar. Then, select “Comm” from the “TARGETS” list, and navigate to the “Signing & Capabilities” tab. You should verify the following settings:
#### Using a Comm development team Apple Developer account
- Team
- Comm Technologies, Inc.
- Bundle Identifier
- app.comm
- Provisioning Profile
- Make sure that the Provisioning Profile exists
#### Using a Personal Apple Developer account
- Team
- Set this to a valid “Team”, which can just be your personal Apple Developer account. “Comm Technologies, Inc.” may be chosen by default, but it’s not valid if you’re using a personal Apple Developer account
- Bundle Identifier
- Pick a unique [Bundle Identifier](https://developer.apple.com/documentation/xcode/preparing-your-app-for-distribution)
- Provisioning Profile
- Make sure that the Provisioning Profile exists
### Building and deploying the app
When you plug your iOS device into your machine for the first time, you’ll be prompted to enter your device passcode to enable debugging and deployment. Click the “Register” button in the dialog that Xcode displays if your device needs to be added to your Provisioning Profile.
Make sure to pull the latest changes and clean the build folder before trying to deploy a build to your device. In Xcode, run Product → Clean Build Folder.
If you’re running a debug build, you’ll need to check that the Metro bundler is running. If you haven’t already, open a new terminal and run:
```
cd native
yarn dev
```
You should finally be ready to build and deploy the app in Xcode! Select your physical device from ”run destinations” in the Workspace Toolbar. Then hit the Run button to build and run the project.
On the launcher screen choose your development server. If this is the first build you’re deploying to the target device, choose the “Enter URL manually” option, and type `http://w.x.y.z:8081`, where `w.x.y.z` is your machine’s local IP address.
Your machine’s local IP address is displayed below the QR code, in the terminal window running the Metro bundler. Try visiting this IP address at port 80, using a browser on your device. It should display an “It works!” message if your iOS device can reach your machine.
If you’re connecting to a local keyserver instance, you’ll want to “Allow Comm to Access” the “Local Network” in your device Settings. This toggle can be found from Settings → Comm. Note that this setting is not enabled by default, and you may have to re-enable it on subsequent build deployments.
### Connecting to local keyserver
If you want your custom build of the app to connect to your local instance of the Node.js server (the `keyserver` subdirectory of the repo), you’ll need to do some additional work. First, confirm that your computer and physical iOS device are on the same network. If you’re running a local keyserver instance, you’ll need to be able to reach it with your device. Local keyservers run on the local IP address at port 3000.
Finally, we need to direct the mobile app to use your local keyserver instance. There are a few different ways to do this, depending on your situation:
- As long as you’re deploying a debug build, your local machine’s IP address should automatically be propagated to the native app when running the Metro bundler. This is the same IP that was discussed in the previous section.
If for whatever reason this value is incorrect, you can override it by setting the `COMM_NAT_DEV_HOSTNAME` environment variable and restarting the Metro bundler:
```sh
cd native
COMM_NAT_DEV_HOSTNAME=w.x.y.z yarn dev
```
Where `w.x.y.z` is the hostname you want to use.
You’ll need to delete and reinstall the app for changes to developer’s machine hostname to take effect, as the default keyserver URL is persisted in Redux.
- If you’re deploying a release build, the above strategy won’t work. Your best bet to override the server URL is to get to the secret “Developer tools” menu option in the app.
1. You may need to use a real production account for this, since the server address will default to the production server if this is the first build you’ve deployed to the target device.
2. Next, in order for the “Developer tools” menu option to appear, you’ll need to add your user ID to [the list of user IDs in `staff.js`](https://github.com/CommE2E/comm/blob/master/lib/facts/staff.js). A good way to figure out your user ID is to use the Chrome Redux debugger to inspect the `currentUserInfo` property when logged into the web app.
3. Finally, you should be able to navigate to Profile → Developer tools in the app and set the address of the local server. It should look something like this:
```
- http://w.x.y.z/comm
+ http://w.x.y.z/webapp
```
Where `w.x.y.z` is the local IP address you found earlier.
- Alternately, if you’re on a release build and option 2 above seems like too much of a hassle, you should be able to simply change the value of [`productionNodeServerURL` in the code](https://github.com/CommE2E/comm/blob/9e6a13f1569787b498a72c890b12ce0dd8323804/native/utils/url-utils.js#L12). Note that you’ll need to delete and reinstall the app for this change to take effect, as the default production URL is persisted in Redux.
## Running Node scripts
To run one of the scripts in `keyserver/src/scripts`, you should start by making sure that the Node server is running. If you haven’t already, open a new terminal and run:
```
cd keyserver
yarn dev
```
Then, from the `keyserver` directory, run `yarn script dist/scripts/name.js`, where `name.js` is the file containing the script.
## Creating a new user
To create a new user, [run the keyserver](#running-keyserver) and the [mobile app on iOS Simulator](#running-mobile-app-on-ios-simulator), and then select “Sign Up” in the app. When you sign up, a new user will be created on your local MariaDB instance. You can then log in on any device connected to your local keyserver.
## Codegen
We use a couple of tools that automatically generate code. There is always a source of truth – usually some file(s) with schemas.
### Codegen for JSI
JSI is a framework in React Native that allows C++ and JS to communicate synchronously and directly. The codegen for JSI takes a Flow schema and generates C++ files that enable communication between JS and C++ in `react-native` apps.
The script to generate this code is written in JavaScript and is included as a npm package so no additional software is needed to use it. The schema has to be defined in Flow as an interface, and that interface must inherit from react-native’s `TurboModule` interface.
To run the JSI codegen, just run:
```
cd native
yarn codegen-jsi
```
The input Flow schemas are located in `native/schema`.
# Debugging
## React Developer Tools
- For web, you can access the React Developer Tools through the Chrome extension by opening the Chrome Developer Tools and selecting the “Components” or “Profiler” tabs. This should work in both our development environment and in production.
- For iOS and Android, you can access the React Developer Tools by running `cd native && yarn react-devtools`. If you want to use it along with the JS debugger (see [Debugging JavaScript](#debugging-javascript)), you should open the JS debugger first, then open React Developer Tools, and finally refresh the app via the dev menu. More details on using React Developer Tools with React Native can be found [here](https://github.com/facebook/react/tree/main/packages/react-devtools#usage-with-react-native).
## Redux Developer Tools
- For web, you can access the Redux Developer Tools through the Chrome extension by opening the Chrome Developer Tools and selecting the “Redux” tab. This should work in both our development environment and in production, although in production you won’t be able to see Redux actions from before you opened up the Redux dev tools.
- For iOS and Android, you can access the Redux Developer Tools by running `cd native && yarn redux-devtools`. On the first open, you’ll need to go to “Settings”, select “Use custom (local) server”, configure it to use port 8043, and then finally hit “Connect”. If you want to use it along with the JS debugger (see [Debugging JavaScript](#debugging-javascript)), you should open the JS debugger first, then open Redux Developer Tools, and finally refresh the app via the dev menu.
## Debugging JavaScript
- For web, you can just use your browser of choice’s dev tools.
- For iOS and Android, you can use the dev menu in dev builds to open up a JS debugger, where you can set breakpoints via the `debugger` expression.
# Working with Phabricator
## Creating a new diff
The biggest difference between GitHub PRs and Phabricator diffs is that a PR corresponds to a branch, whereas a diff corresponds to a commit.
When you have a commit ready and want to submit it for code review, just run `arc diff` from within the Comm Git repo. `arc diff` will look at the most recent commit in `git log` and create a new diff for it.
## Updating a diff
With GitHub PRs, updates are usually performed by adding on more commits. In contrast, in Phabricator a diff is updated by simply amending the existing commit and running `arc diff` again.
When you run `arc diff`, it looks for a `Differential Revision: ` line in the commit text of the most recent commit. If Arcanist finds that line, it will assume you want to update the existing diff rather than create a new one. Other Arcanist commands such as `arc amend` (which amends commit text to match a diff on Phabricator) also look for the `Differential Revision: ` line.
## Working with a stack
One of the advantages of Phabricator’s approach is that larger, multi-part changes can be split up into smaller pieces for review. These multi-part changes are usually referred to as a “stack” of diffs.
When creating a diff that depends on another, you should make sure to create a dependency relationship between those two diffs, so that your reviewers can see the stack on Phabricator. The easiest way to do that is to include `Depends on D123` in the commit text of the child commit, but the dependency relationship can also be specified using the Phabricator web UI.
You’ll find that mastering Git’s interactive rebase feature (`git rebase -i`) will help you a lot when working with stacks. Interactive rebases make it easy to “diff up” multiple commits at once, or to amend a specific commit in the middle of a stack in response to a review.
## Committing a diff
After your diff has been accepted, you should be able to land it. To land a diff just run `arc land` from within the repository.
If you have a stack of unlanded commits in your Git branch, `arc land` will attempt to land all of those diffs. If some of the diffs in your stack haven’t been accepted yet, you’ll need to create a new, separate branch that contains just the commits you want to land before running `arc land`.
Note that you need commit rights to the repository in order to run `arc land`. If you don’t have commit rights, reach out to @ashoat for assistance.
## Final notes
When developing, I usually just pop up three terminal windows, one for `yarn dev` in each of keyserver, web, and native.
Note that it’s currently only possible to create a user account using the iOS or Android apps. The website supports logging in, but does not support account creation.
Good luck, and let @ashoat know if you have any questions!
diff --git a/docs/linux_dev_environment.md b/docs/linux_dev_environment.md
index cac19924a..d90227129 100644
--- a/docs/linux_dev_environment.md
+++ b/docs/linux_dev_environment.md
@@ -1,144 +1,145 @@
# Linux Instructions
The instructions to set up Comm development environment are only tested and available for Ubuntu 20.04.
## Reference main dev environment instructions
This doc is very incomplete, so you should start by reading through the [main dev environment instructions](dev_environment.md). For anything that isn't mentioned here, you should either try to install it with the same approach described in the main docs, or (if that's not possible) you should try to figure out an alternate approach yourself. The following notes can help you for some of the more complex parts.
## React Native
To set up React Native, follow the instructions in the [React Native docs](https://reactnative.dev/docs/environment-setup).
## nvm
You can install nvm following the [nvm installation instructions](https://github.com/nvm-sh/nvm#installing-and-updating).
## Redis
On Ubuntu, you can install the latest stable version of Redis from the `redislabs/redis` package repository via `apt`:
```
sudo add-apt-repository ppa:redislabs/redis
sudo apt-get update
sudo apt-get install redis
sudo systemctl start redis
sudo systemctl status redis
```
If you’re not on Ubuntu, you can look for distro-specific binaries using your preferred package manager, or compile from [source](https://redis.io/download).
## Reactotron
Reactotron is an event tracker and logger that can be used to aid in debugging on React Native.
Download and install the Reactotron package from their [release](https://github.com/infinitered/reactotron/blob/master/docs/installing.md) page.
If Reactotron does not connect to the Android Emulator, run the following command and restart the Metro bundler:
```
adb reverse tcp:9090 tcp:9090
```
You can see more information on the [Reactotron installation instructions](https://github.com/infinitered/reactotron/blob/master/docs/quick-start-react-native.md#configure-reactotron-with-your-project).
## phpMyAdmin
On Ubuntu, you can install and run `phpmyadmin` using `apt`:
```
sudo apt install phpmyadmin php-mbstring php-zip php-gd php-json php-curl
phpmyadmin
```
For configuration you should refer to the [main dev environment instructions](dev_environment.md#phpmyadmin).
## Apache
On Ubuntu, you can install the Apache webserver using `apt`:
```
sudo apt install apache2
```
The you can enable it to run on system startup using `systemd`:
```
sudo systemctl start apache2
sudo systemctl status apache2
```
You'll also need to add a set of Apache modules:
```
sudo a2ensite mod_proxy mod_proxy_http mod_proxy_wstunnel mod_userdir
```
Finally, you'll need to configure the development site. Open `000-default.conf` with your text editor of choice:
```
sudo vim /etc/apache2/sites-enabled/000-default.conf
```
Add the content below, but make sure to replace “ashoat” with your username.
```
AllowOverride All
Options Indexes FollowSymLinks
Require all granted
ProxyRequests on
- ProxyPass /comm/ws ws://localhost:3000/ws
- ProxyPass /comm/ http://localhost:3000/
+ ProxyPass /keyserver/ http://localhost:3000/keyserver/
+ ProxyPass /keyserver/ws ws://localhost:3000/keyserver/ws
+ ProxyPass /webapp/ http://localhost:3000/webapp/
ProxyPass /commlanding/ http://localhost:3000/commlanding/
RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME}
RequestHeader set "X-Forwarded-SSL" expr=%{HTTPS}
```
Finally, let’s restart Apache so it picks up the changes.
```
sudo systemctl restart apache2
sudo systemctl status apache2
```
## MySQL
First, install Docker with the following commands:
```
sudo apt-get remove docker docker-engine docker.io
sudo apt install docker.io
sudo systemctl start docker
sudo systemctl enable docker
```
Next you'll download the MySQL 5.7 Docker image:
```
docker pull mysql:5.7
```
After setting up your MySQL install following the [main dev environment instructions](dev_environment.md#mysql-2), you can use the root password to run your Docker container:
```
docker run --name my-mysql -e MYSQL_ROOT_PASSWORD=password --expose 3306 -p 3306:3306 -v $HOME/mysql-data:/var/lib/mysql -d mysql:5.7
```
You can log in to MySQL database in the Docker container using the following command:
```
docker exec -it my-mysql mysql -p
```
You may find yourself needing to configure your MySQL user using the IP corresponding to the inet entry from `ifconfig docker0` output, rather than the default (`127.0.0.1`):
```
CREATE USER comm@172.17.0.1 IDENTIFIED BY 'password';
GRANT ALL ON comm.* TO comm@172.17.0.1;
```
diff --git a/docs/nix_keyserver_deployment.md b/docs/nix_keyserver_deployment.md
index c66e5dfe7..eeca5b063 100644
--- a/docs/nix_keyserver_deployment.md
+++ b/docs/nix_keyserver_deployment.md
@@ -1,71 +1,72 @@
# Keyserver Deployment
Deploying the keyserver requires configuring it, building its Docker image, and deploying that image with Docker Compose.
## Configuration
In order for the keyserver to interface with dependencies, host the landing page, and host the Comm web application, the following must be added to `keyserver/.env`:
```
# Mandatory
COMM_DATABASE_DATABASE=comm
COMM_DATABASE_USER=
COMM_DATABASE_PASSWORD=
COMM_JSONCONFIG_secrets_user_credentials='{"username":"","password":""}'
COMM_JSONCONFIG_facts_landing_url='{"baseDomain":"http://localhost","basePath":"/commlanding/","baseRoutePath":"/commlanding/","https":false}'
-COMM_JSONCONFIG_facts_commapp_url='{"baseDomain":"http://localhost:3000","basePath":"/comm/","https":false,"baseRoutePath":"/comm/","proxy":"none"}'
+COMM_JSONCONFIG_facts_webapp_url='{"baseDomain":"http://localhost:3000","basePath":"/webapp/","https":false,"baseRoutePath":"/webapp/","proxy":"none"}'
+COMM_JSONCONFIG_facts_keyserver_url='{"baseDomain":"http://localhost:3000","basePath":"/keyserver/","baseRoutePath":"/keyserver/","https":false,"proxy":"none"}'
COMM_JSONCONFIG_facts_webapp_cors='{"domain": "http://localhost:3000"}'
# Required to connect to production Identity service
COMM_JSONCONFIG_secrets_identity_service_config="{\"identitySocketAddr\":\"https://identity.commtechnologies.org:50054\"}"
# Required for ETH login
COMM_JSONCONFIG_secrets_alchemy='{"key":""}'
COMM_JSONCONFIG_secrets_walletconnect='{"key":""}'
# Example backup configuration that stores up to 10 GiB of backups in /home/comm/backups
COMM_JSONCONFIG_facts_backups='{"enabled":true,"directory":"/home/comm/backups","maxDirSizeMiB":10240}'
```
### MariaDB configuration
- `COMM_DATABASE_DATABASE`: Specifies the name of the database the keyserver will use.
- `COMM_DATABASE_USER`: The username the keyserver uses to connect to MariaDB. Replace `` with your desired username.
- `COMM_DATABASE_PASSWORD`: Corresponding password for the above user. Replace `` with your desired password.
### Identity service configuration
- `COMM_JSONCONFIG_secrets_user_credentials`: Credentials for authenticating against the Identity service. Replace `` and `` with any values. In the future, they will need to be actual credentials registered with the Identity service.
- `COMM_JSONCONFIG_secrets_identity_service_config`: Socket address for the Identity service. If omitted, the keyserver will try to connect to a local instance of the Identity service.
### ETH login configuration
- `COMM_JSONCONFIG_secrets_alchemy`: Alchemy key used for Ethereum Name Service (ENS) resolution and retrieving ETH public keys. Replace `` with your actual key.
- `COMM_JSONCONFIG_secrets_walletconnect`: WalletConnect key used to enable Sign-In with Ethereum (SIWE). Replace `` with your actual key.
### URL configuration
-- `COMM_JSONCONFIG_facts_commapp_url`: Your keyserver needs to know what its externally-facing URL is in order to construct links. It also needs to know if it’s being proxied to that externally-facing URL, and what the internal route path is.
+- `COMM_JSONCONFIG_facts_keyserver_url`: Your keyserver needs to know what its externally-facing URL is in order to construct links. It also needs to know if it’s being proxied to that externally-facing URL, and what the internal route path is.
- `baseDomain`: Externally-facing domain. Used for constructing links.
- `basePath`: Externally-facing path. Used for constructing links.
- `baseRoutePath`: Internally-facing path. Same as basePath if no proxy. If there’s a proxy, this is the local path (e.g. http://localhost:3000/landing would correspond with /landing/)
- `proxy`: `"none" | "apache"` Determines which request headers to use for HTTPS validation and IP address timezone detection.
- `https`: If true, checks request headers to validate that HTTPS is in use.
### CORS configuration
- `COMM_JSONCONFIG_facts_webapp_cors`: Your keyserver needs to be able to include CORS headers with the domain where the comm web application is hosted.
- `domain`: Domain where the web application is hosted.
### Backup configuration
- `COMM_JSONCONFIG_facts_backups`: Specifies whether to enable backups, where to store them, and the max size of the backups directory.
## Building & deploying
Once configured, the keyserver can be built and deployed by simply running:
```bash
cd keyserver
./bash/dc.sh up --build
```
diff --git a/docs/nix_web_workflows.md b/docs/nix_web_workflows.md
index 805698643..0d3ae155b 100644
--- a/docs/nix_web_workflows.md
+++ b/docs/nix_web_workflows.md
@@ -1,81 +1,81 @@
# Development
## Flow typechecker
It’s good to run the `flow` typechecker frequently to make sure you’re not introducing any type errors. Flow treats each Yarn Workspace as a separate environment, and as such runs a separate type-checking server for each.
You should now be able to run `flow` in any of the Yarn workspaces:
```
cd lib
flow
```
## Running keyserver
To run the web app or the landing page, you’ll first need to run the keyserver locally.
Open a new terminal and run:
```
cd keyserver
yarn dev
```
This command runs three processes. The first two are to keep the `dist` folder updated whenever the `src` folder changes. They are “watch” versions of the same Babel and `rsync` commands we used to initially create the `dist` folder (before running the `generate-olm-config.js` script above). The final process is `nodemon`, which is similar to `node` except that it restarts whenever any of its source files (in the `dist` directory) changes.
Note that if you run `yarn dev` in `keyserver` right after `yarn cleaninstall`, before Webpack is given a chance to build `app.build.cjs`/`landing.build.cjs` files, then Node will crash when it attempts to import those files. Just make sure to run `yarn dev` (or `yarn prod`) in `web` or `landing` before attempting to load the corresponding webpages.
## Running web app
First, make sure that the keyserver is running. If you haven’t already, run:
```
cd keyserver
yarn dev
```
Next, open a new terminal and run:
```
cd web
yarn dev
```
-You should now be able to load the web app in your web browser at http://localhost:3000/comm/.
+You should now be able to load the web app in your web browser at http://localhost:3000/webapp/.
This command will start two processes. One is `webpack-dev-server`, which will serve the JS files. `webpack-dev-server` also makes sure the website automatically hot-reloads whenever any of the source files change. The other process is `webpack --watch`, which will build the `app.build.cjs` file, as well as rebuilding it whenever any of the source files change. The `app.build.cjs` file is consumed by the Node server in order to pre-render the initial HTML from the web source (“Server-Side Rendering”).
## Running landing page
First, make sure that the keyserver is running. If you haven’t already, run:
```
cd keyserver
yarn dev
```
Next, open a new terminal and run:
```
cd landing
yarn dev
```
You should now be able to load the landing page in your web browser at http://localhost:3000/commlanding/.
This runs the same two processes as the web app, but for the landing page. Note that the `landing.build.cjs` file (similar to the web app’s `app.build.cjs` file) is consumed by the Node server.
## Debugging
### React Developer Tools
You can access the React Developer Tools through the Chrome extension by opening the Chrome Developer Tools and selecting the “Components” or “Profiler” tabs. This should work in both our development environment and in production.
### Redux Developer Tools
You can access the Redux Developer Tools through the Chrome extension by opening the Chrome Developer Tools and selecting the “Redux” tab. This should work in both our development environment and in production, although in production you won’t be able to see Redux actions from before you opened up the Redux dev tools.
### Debugging JavaScript
You can just use your browser of choice’s dev tools.
diff --git a/keyserver/icons/site.webmanifest b/keyserver/icons/site.webmanifest
index 57505d48f..b00aef7e6 100644
--- a/keyserver/icons/site.webmanifest
+++ b/keyserver/icons/site.webmanifest
@@ -1,19 +1,19 @@
{
"name": "SquadCal",
"short_name": "SquadCal",
"icons": [
{
- "src": "/android-chrome-192x192.png",
+ "src": "android-chrome-192x192.png",
"sizes": "192x192",
"type": "image/png"
},
{
- "src": "/android-chrome-512x512.png",
+ "src": "android-chrome-512x512.png",
"sizes": "512x512",
"type": "image/png"
}
],
"theme_color": "#b91d47",
"background_color": "#b91d47",
"display": "standalone"
}
diff --git a/keyserver/src/creators/report-creator.js b/keyserver/src/creators/report-creator.js
index 60ce12b8a..e6e7658db 100644
--- a/keyserver/src/creators/report-creator.js
+++ b/keyserver/src/creators/report-creator.js
@@ -1,238 +1,238 @@
// @flow
import _isEqual from 'lodash/fp/isEqual.js';
import bots from 'lib/facts/bots.js';
import {
filterRawEntryInfosByCalendarQuery,
serverEntryInfosObject,
} from 'lib/shared/entry-utils.js';
import { messageTypes } from 'lib/types/message-types-enum.js';
import {
type ReportCreationRequest,
type ReportCreationResponse,
type ThreadInconsistencyReportCreationRequest,
type EntryInconsistencyReportCreationRequest,
type UserInconsistencyReportCreationRequest,
reportTypes,
} from 'lib/types/report-types.js';
import { values } from 'lib/utils/objects.js';
import {
sanitizeReduxReport,
type ReduxCrashReport,
} from 'lib/utils/sanitization.js';
import createIDs from './id-creator.js';
import createMessages from './message-creator.js';
import { dbQuery, SQL } from '../database/database.js';
import { fetchUsername } from '../fetchers/user-fetchers.js';
import { handleAsyncPromise } from '../responders/handlers.js';
import { createBotViewer } from '../session/bots.js';
import type { Viewer } from '../session/viewer.js';
-import { getAndAssertCommAppURLFacts } from '../utils/urls.js';
+import { getAndAssertKeyserverURLFacts } from '../utils/urls.js';
const { commbot } = bots;
async function createReport(
viewer: Viewer,
request: ReportCreationRequest,
): Promise {
const shouldIgnore = await ignoreReport(viewer, request);
if (shouldIgnore) {
return null;
}
const [id] = await createIDs('reports', 1);
let type, report, time;
if (request.type === reportTypes.THREAD_INCONSISTENCY) {
({ type, time, ...report } = request);
time = time ? time : Date.now();
} else if (request.type === reportTypes.ENTRY_INCONSISTENCY) {
({ type, time, ...report } = request);
} else if (request.type === reportTypes.MEDIA_MISSION) {
({ type, time, ...report } = request);
} else if (request.type === reportTypes.USER_INCONSISTENCY) {
({ type, time, ...report } = request);
} else {
({ type, ...report } = request);
time = Date.now();
const redactedReduxReport: ReduxCrashReport = sanitizeReduxReport({
preloadedState: report.preloadedState,
currentState: report.currentState,
actions: report.actions,
});
report = {
...report,
...redactedReduxReport,
};
}
const row = [
id,
viewer.id,
type,
request.platformDetails.platform,
JSON.stringify(report),
time,
];
const query = SQL`
INSERT INTO reports (id, user, type, platform, report, creation_time)
VALUES ${[row]}
`;
await dbQuery(query);
handleAsyncPromise(sendCommbotMessage(viewer, request, id));
return { id };
}
async function sendCommbotMessage(
viewer: Viewer,
request: ReportCreationRequest,
reportID: string,
): Promise {
const canGenerateMessage = getCommbotMessage(request, reportID, null);
if (!canGenerateMessage) {
return;
}
const username = await fetchUsername(viewer.id);
const message = getCommbotMessage(request, reportID, username);
if (!message) {
return;
}
const time = Date.now();
await createMessages(createBotViewer(commbot.userID), [
{
type: messageTypes.TEXT,
threadID: commbot.staffThreadID,
creatorID: commbot.userID,
time,
text: message,
},
]);
}
async function ignoreReport(
viewer: Viewer,
request: ReportCreationRequest,
): Promise {
// The below logic is to avoid duplicate inconsistency reports
if (
request.type !== reportTypes.THREAD_INCONSISTENCY &&
request.type !== reportTypes.ENTRY_INCONSISTENCY
) {
return false;
}
const { type, platformDetails, time } = request;
if (!time) {
return false;
}
const { platform } = platformDetails;
const query = SQL`
SELECT id
FROM reports
WHERE user = ${viewer.id} AND type = ${type}
AND platform = ${platform} AND creation_time = ${time}
`;
const [result] = await dbQuery(query);
return result.length !== 0;
}
function getCommbotMessage(
request: ReportCreationRequest,
reportID: string,
username: ?string,
): ?string {
const name = username ? username : '[null]';
const { platformDetails } = request;
const { platform, codeVersion } = platformDetails;
const platformString = codeVersion ? `${platform} v${codeVersion}` : platform;
if (request.type === reportTypes.ERROR) {
- const { baseDomain, basePath } = getAndAssertCommAppURLFacts();
+ const { baseDomain, basePath } = getAndAssertKeyserverURLFacts();
return (
`${name} got an error :(\n` +
`using ${platformString}\n` +
`${baseDomain}${basePath}download_error_report/${reportID}`
);
} else if (request.type === reportTypes.THREAD_INCONSISTENCY) {
const nonMatchingThreadIDs = getInconsistentThreadIDsFromReport(request);
const nonMatchingString = [...nonMatchingThreadIDs].join(', ');
return (
`system detected inconsistency for ${name}!\n` +
`using ${platformString}\n` +
`occurred during ${request.action.type}\n` +
`thread IDs that are inconsistent: ${nonMatchingString}`
);
} else if (request.type === reportTypes.ENTRY_INCONSISTENCY) {
const nonMatchingEntryIDs = getInconsistentEntryIDsFromReport(request);
const nonMatchingString = [...nonMatchingEntryIDs].join(', ');
return (
`system detected inconsistency for ${name}!\n` +
`using ${platformString}\n` +
`occurred during ${request.action.type}\n` +
`entry IDs that are inconsistent: ${nonMatchingString}`
);
} else if (request.type === reportTypes.USER_INCONSISTENCY) {
const nonMatchingUserIDs = getInconsistentUserIDsFromReport(request);
const nonMatchingString = [...nonMatchingUserIDs].join(', ');
return (
`system detected inconsistency for ${name}!\n` +
`using ${platformString}\n` +
`occurred during ${request.action.type}\n` +
`user IDs that are inconsistent: ${nonMatchingString}`
);
} else if (request.type === reportTypes.MEDIA_MISSION) {
const mediaMissionJSON = JSON.stringify(request.mediaMission);
const success = request.mediaMission.result.success
? 'media mission success!'
: 'media mission failed :(';
return `${name} ${success}\n` + mediaMissionJSON;
} else {
return null;
}
}
function findInconsistentObjectKeys(
first: { +[id: string]: O },
second: { +[id: string]: O },
): Set {
const nonMatchingIDs = new Set();
for (const id in first) {
if (!_isEqual(first[id])(second[id])) {
nonMatchingIDs.add(id);
}
}
for (const id in second) {
if (!first[id]) {
nonMatchingIDs.add(id);
}
}
return nonMatchingIDs;
}
function getInconsistentThreadIDsFromReport(
request: ThreadInconsistencyReportCreationRequest,
): Set {
const { pushResult, beforeAction } = request;
return findInconsistentObjectKeys(beforeAction, pushResult);
}
function getInconsistentEntryIDsFromReport(
request: EntryInconsistencyReportCreationRequest,
): Set {
const { pushResult, beforeAction, calendarQuery } = request;
const filteredBeforeAction = filterRawEntryInfosByCalendarQuery(
serverEntryInfosObject(values(beforeAction)),
calendarQuery,
);
const filteredAfterAction = filterRawEntryInfosByCalendarQuery(
serverEntryInfosObject(values(pushResult)),
calendarQuery,
);
return findInconsistentObjectKeys(filteredBeforeAction, filteredAfterAction);
}
function getInconsistentUserIDsFromReport(
request: UserInconsistencyReportCreationRequest,
): Set {
const { beforeStateCheck, afterStateCheck } = request;
return findInconsistentObjectKeys(beforeStateCheck, afterStateCheck);
}
export default createReport;
diff --git a/keyserver/src/fetchers/upload-fetchers.js b/keyserver/src/fetchers/upload-fetchers.js
index 2a1ff20a0..fd8e964ee 100644
--- a/keyserver/src/fetchers/upload-fetchers.js
+++ b/keyserver/src/fetchers/upload-fetchers.js
@@ -1,408 +1,408 @@
// @flow
import ip from 'internal-ip';
import _keyBy from 'lodash/fp/keyBy.js';
import type { Media, Image, EncryptedImage } from 'lib/types/media-types.js';
import { messageTypes } from 'lib/types/message-types-enum.js';
import type { MediaMessageServerDBContent } from 'lib/types/messages/media.js';
import { getUploadIDsFromMediaMessageServerDBContents } from 'lib/types/messages/media.js';
import { threadPermissions } from 'lib/types/thread-permission-types.js';
import type {
ThreadFetchMediaResult,
ThreadFetchMediaRequest,
} from 'lib/types/thread-types.js';
import { makeBlobServiceURI } from 'lib/utils/blob-service.js';
import { isDev } from 'lib/utils/dev-utils.js';
import { ServerError } from 'lib/utils/errors.js';
import { dbQuery, SQL } from '../database/database.js';
import type { Viewer } from '../session/viewer.js';
-import { getAndAssertCommAppURLFacts } from '../utils/urls.js';
+import { getAndAssertKeyserverURLFacts } from '../utils/urls.js';
type UploadInfo = {
content: Buffer,
mime: string,
};
async function fetchUpload(
viewer: Viewer,
id: string,
secret: string,
): Promise {
const query = SQL`
SELECT content, mime, extra
FROM uploads
WHERE id = ${id} AND secret = ${secret}
`;
const [result] = await dbQuery(query);
if (result.length === 0) {
throw new ServerError('invalid_parameters');
}
const [row] = result;
const { content, mime, extra } = row;
const { blobHash } = JSON.parse(extra);
if (blobHash) {
throw new ServerError('resource_unavailable');
}
return { content, mime };
}
async function fetchUploadChunk(
id: string,
secret: string,
pos: number,
len: number,
): Promise {
// We use pos + 1 because SQL is 1-indexed whereas js is 0-indexed
const query = SQL`
SELECT SUBSTRING(content, ${pos + 1}, ${len}) AS content, mime, extra
FROM uploads
WHERE id = ${id} AND secret = ${secret}
`;
const [result] = await dbQuery(query);
if (result.length === 0) {
throw new ServerError('invalid_parameters');
}
const [row] = result;
const { content, mime, extra } = row;
if (extra) {
const { blobHash } = JSON.parse(extra);
if (blobHash) {
throw new ServerError('resource_unavailable');
}
}
return {
content,
mime,
};
}
// Returns total size in bytes.
async function getUploadSize(id: string, secret: string): Promise {
const query = SQL`
SELECT LENGTH(content) AS length, extra
FROM uploads
WHERE id = ${id} AND secret = ${secret}
`;
const [result] = await dbQuery(query);
if (result.length === 0) {
throw new ServerError('invalid_parameters');
}
const [row] = result;
const { length, extra } = row;
if (extra) {
const { blobHash } = JSON.parse(extra);
if (blobHash) {
throw new ServerError('resource_unavailable');
}
}
return length;
}
function getUploadURL(id: string, secret: string): string {
- const { baseDomain, basePath } = getAndAssertCommAppURLFacts();
+ const { baseDomain, basePath } = getAndAssertKeyserverURLFacts();
const uploadPath = `${basePath}upload/${id}/${secret}`;
if (isDev) {
const ipV4 = ip.v4.sync() || 'localhost';
const port = parseInt(process.env.PORT, 10) || 3000;
return `http://${ipV4}:${port}${uploadPath}`;
}
return `${baseDomain}${uploadPath}`;
}
function makeUploadURI(blobHash: ?string, id: string, secret: string): string {
if (blobHash) {
return makeBlobServiceURI(blobHash);
}
return getUploadURL(id, secret);
}
function imagesFromRow(row: Object): Image | EncryptedImage {
const uploadExtra = JSON.parse(row.uploadExtra);
const { width, height, blobHash, thumbHash } = uploadExtra;
const { uploadType: type, uploadSecret: secret } = row;
const id = row.uploadID.toString();
const dimensions = { width, height };
const uri = makeUploadURI(blobHash, id, secret);
const isEncrypted = !!uploadExtra.encryptionKey;
if (type !== 'photo') {
throw new ServerError('invalid_parameters');
}
if (!isEncrypted) {
return { id, type: 'photo', uri, dimensions, thumbHash };
}
return {
id,
type: 'encrypted_photo',
blobURI: uri,
dimensions,
thumbHash,
encryptionKey: uploadExtra.encryptionKey,
};
}
async function fetchImages(
viewer: Viewer,
mediaIDs: $ReadOnlyArray,
): Promise<$ReadOnlyArray> {
const query = SQL`
SELECT id AS uploadID, secret AS uploadSecret,
type AS uploadType, extra AS uploadExtra
FROM uploads
WHERE id IN (${mediaIDs}) AND uploader = ${viewer.id} AND container IS NULL
`;
const [result] = await dbQuery(query);
return result.map(imagesFromRow);
}
async function fetchMediaForThread(
viewer: Viewer,
request: ThreadFetchMediaRequest,
): Promise {
const visibleExtractString = `$.${threadPermissions.VISIBLE}.value`;
const query = SQL`
SELECT content.photo AS uploadID,
u.secret AS uploadSecret,
u.type AS uploadType, u.extra AS uploadExtra,
u.container, u.creation_time,
NULL AS thumbnailID,
NULL AS thumbnailUploadSecret,
NULL AS thumbnailUploadExtra
FROM messages m
LEFT JOIN JSON_TABLE(
m.content,
"$[*]" COLUMNS(photo INT PATH "$")
) content ON 1
LEFT JOIN uploads u ON u.id = content.photo
LEFT JOIN memberships mm ON mm.thread = ${request.threadID}
AND mm.user = ${viewer.id}
WHERE m.thread = ${request.threadID} AND m.type = ${messageTypes.IMAGES}
AND JSON_EXTRACT(mm.permissions, ${visibleExtractString}) IS TRUE
UNION SELECT content.media AS uploadID,
uv.secret AS uploadSecret,
uv.type AS uploadType, uv.extra AS uploadExtra,
uv.container, uv.creation_time,
content.thumbnail AS thumbnailID,
ut.secret AS thumbnailUploadSecret,
ut.extra AS thumbnailUploadExtra
FROM messages m
LEFT JOIN JSON_TABLE(
m.content,
"$[*]" COLUMNS(
media INT PATH "$.uploadID",
thumbnail INT PATH "$.thumbnailUploadID"
)
) content ON 1
LEFT JOIN uploads uv ON uv.id = content.media
LEFT JOIN uploads ut ON ut.id = content.thumbnail
LEFT JOIN memberships mm ON mm.thread = ${request.threadID}
AND mm.user = ${viewer.id}
WHERE m.thread = ${request.threadID}
AND m.type = ${messageTypes.MULTIMEDIA}
AND JSON_EXTRACT(mm.permissions, ${visibleExtractString}) IS TRUE
ORDER BY creation_time DESC
LIMIT ${request.limit} OFFSET ${request.offset}
`;
const [uploads] = await dbQuery(query);
const media = uploads.map(upload => {
const { uploadID, uploadType, uploadSecret, uploadExtra } = upload;
const { width, height, encryptionKey, blobHash, thumbHash } =
JSON.parse(uploadExtra);
const dimensions = { width, height };
const uri = makeUploadURI(blobHash, uploadID, uploadSecret);
if (uploadType === 'photo') {
if (encryptionKey) {
return {
type: 'encrypted_photo',
id: uploadID.toString(),
blobURI: uri,
encryptionKey,
dimensions,
thumbHash,
};
}
return {
type: 'photo',
id: uploadID.toString(),
uri,
dimensions,
thumbHash,
};
}
const { thumbnailUploadSecret, thumbnailUploadExtra } = upload;
const thumbnailID = upload.thumbnailID?.toString();
const {
encryptionKey: thumbnailEncryptionKey,
blobHash: thumbnailBlobHash,
thumbHash: thumbnailThumbHash,
} = JSON.parse(thumbnailUploadExtra);
const thumbnailURI = makeUploadURI(
thumbnailBlobHash,
thumbnailID,
thumbnailUploadSecret,
);
if (encryptionKey) {
return {
type: 'encrypted_video',
id: uploadID.toString(),
blobURI: uri,
encryptionKey,
dimensions,
thumbnailID,
thumbnailBlobURI: thumbnailURI,
thumbnailEncryptionKey,
thumbnailThumbHash,
};
}
return {
type: 'video',
id: uploadID.toString(),
uri,
dimensions,
thumbnailID,
thumbnailURI,
thumbnailThumbHash,
};
});
return { media };
}
async function fetchUploadsForMessage(
viewer: Viewer,
mediaMessageContents: $ReadOnlyArray,
): Promise<$ReadOnlyArray