diff --git a/docs/dev_environment.md b/docs/dev_environment.md index 68c11d2df..efeaa04fd 100644 --- a/docs/dev_environment.md +++ b/docs/dev_environment.md @@ -1,588 +1,601 @@ # Requirements Please note that our dev environment currently only works on macOS.
Why not Windows or Linux? (click to expand)

It’s primarily because Apple only supports iOS development using macOS. It’s true that we could support web, server, and Android development on other operating systems, but because of the Apple requirement, all of our active developers currently run macOS. We’d very much welcome a PR to build out support on Windows or Linux!

Unfortunately the dev environment is overall pretty heavy. You’ll ideally want a machine with at least 32 GiB of RAM, although 16 GiB should suffice. # Prerequisites ## Xcode Go to the [Mac App Store](https://apps.apple.com/us/app/xcode/id497799835) to install Xcode, or if you already have it, to update it to the latest version. Once Xcode is installed, open it up. If you are prompted, follow the instructions to install any “Additional Required Components”. Finally, you need to make sure that the “Command Line Tools” are installed. Go to Xcode → Preferences → Locations, and then install the tools by selecting the most recent version from the Command Line Tools dropdown. ## Homebrew Install [Homebrew](https://brew.sh/), a package manager for macOS. ## Node Next, install [Node](https://nodejs.org/) using Homebrew. ``` brew install node; brew upgrade node ``` The reason we use both `install` and `upgrade` is that there’s no single Homebrew command equivalent to “install if not installed, and upgrade if already installed”. ## Yarn We use the [Yarn](https://yarnpkg.com/) package manager for JavaScript in our repo. ``` brew install yarn; brew upgrade yarn ``` ## Watchman Watchman is a tool from Facebook used in the React Native dev environment to watch for changes to your filesystem. ``` brew install watchman; brew upgrade watchman ``` ## nvm Node Version Manager is a tool that helps us make sure we use the same version of Node on our server between prod and dev environments. ``` brew install nvm; brew upgrade nvm ``` After installing, Homebrew will print out some instructions under the Caveats section of its output. It will ask you to do two things: `mkdir ~/.nvm`, and to add some lines to your `~/.bash_profile` (or desired shell configuration file). We recommend that you append `--no-use` to the line that loads nvm, so that you continue to use your Homebrew-sourced Node distribution by default: ``` export NVM_DIR="$HOME/.nvm" [ -s "/usr/local/opt/nvm/nvm.sh" ] && . "/usr/local/opt/nvm/nvm.sh" --no-use # This loads nvm [ -s "/usr/local/opt/nvm/etc/bash_completion.d/nvm" ] && . "/usr/local/opt/nvm/etc/bash_completion.d/nvm" # This loads nvm bash_completion ``` Now either close and reopen your terminal window or re-source your shell configuration file in order to load nvm: ``` source ~/.bash_profile ``` ## MySQL For now we’re using MySQL 5.7 as the primary server-side database. Hopefully we’ll change this soon, but for now, install MySQL 5.7 using Homebrew. ``` brew install mysql@5.7; brew upgrade mysql@5.7 ``` Next we’ll configure MySQL to start when your computer boots using `brew services`: ``` brew tap homebrew/services brew services start mysql@5.7 ``` We’ll also want to link MySQL so that you can run CLI commands: ``` brew link mysql@5.7 --force ``` Finally, you should set up a root password for your local MySQL instance: ``` mysqladmin -u root password ``` ## Redis We use Redis on the server side as a message broker. ``` brew install redis; brew upgrade redis ``` We’ll set it up to start on boot with `brew services`: ``` brew services start redis ``` ## CocoaPods CocoaPods is a dependency management system for iOS development. React Native uses it to manage native modules. ``` sudo gem install cocoapods ``` In order for `pod` to be accessible from the command-line, we’ll need to update your `$PATH` environmental variable. Open your `~/.bash_profile` (or desired shell configuration file) and add the following line: ``` export PATH=$PATH:/usr/local/lib/ruby/gems/2.7.0/bin ``` Make sure you reload the `~/.bash_profile` after editing it: ``` source ~/.bash_profile ``` ## React Native Debugger The React Native Debugger allows you to step through Javascript execution, track Redux state, and inspect the React component tree. ``` brew install react-native-debugger; brew upgrade react-native-debugger ``` ## Reactotron Reactotron is an event tracker and logger that can be used to aid in debugging on React Native. ``` brew install reactotron; brew upgrade reactotron ``` ## JDK We’ll need the Java Development Kit for Android development. ``` brew install adoptopenjdk/openjdk/adoptopenjdk8; brew upgrade adoptopenjdk/openjdk/adoptopenjdk8 ``` ## Android Studio Start by downloading and installing [Android Studio](https://developer.android.com/studio/index.html). When prompted to choose an installation type, select “Custom”. Make sure you check the boxes for the following: * `Android SDK` * `Android SDK Platform` * `Performance (Intel ® HAXM)` * `Android Virtual Device` ### Android SDK Android Studio installs the latest Android SDK by default, but since React Native uses the Android 10 SDK specifically, we’ll need to install it using Android Studio’s SDK Manager. You can access the SDK Manager from the “Welcome to Android Studio” screen that pops up when you first open the application, under “Configure”. If you already have a project open, you can access it from Tools → SDK Manager. Once you have the SDK Manager open, select the “SDK Platforms” tab, and then check the box for “Show Package Details”. Now expand the “Android 10 (Q)” section, and make sure the following subsections are checked: * `Android SDK Platform 29` * `Intel x86 Atom_64 System Image` or `Google APIs Intel x86 Atom System Image` Next, select the “SDK Tools” tab, and check the box for “Show Package Details”. Expand the “Android SDK Build-Tools” section, and make sure that the “29.0.2” subsection is checked. To finish the SDK Manager step, click “Apply” to download and install everything you’ve selected. ### Enable Android CLI commands You’ll need to append the following lines to your `~/.bash_profile` (or desired shell configuration file) in order for React Native to be able to build your Android project. ``` export ANDROID_HOME=$HOME/Library/Android/sdk export PATH=$PATH:$ANDROID_HOME/emulator export PATH=$PATH:$ANDROID_HOME/tools export PATH=$PATH:$ANDROID_HOME/tools/bin export PATH=$PATH:$ANDROID_HOME/platform-tools export JAVA_HOME="/Applications/Android Studio.app/Contents/jre/jdk/Contents/Home" ``` Now either close and reopen your terminal window or re-source your shell configuration file in order to run the new commands: ``` source ~/.bash_profile ``` ## Arcanist We use Phabricator for code review. To upload a “diff” to Phabricator, you’ll need to use a tool called Arcanist. To install Arcanist, we’ll need to clone its Git repository. Pick a place in your filesystem to store it, and then run this command: ``` git clone https://github.com/phacility/arcanist.git ``` Next, you’ll need to add the path `./arcanist/bin` to your `$PATH` in your `~/.bash_profile` (or desired shell configuration file): ``` export PATH=$PATH:~/src/arcanist/bin ``` Make sure to replace the `~/src` portion of the above with the location of the directory you installed Arcanist in. # Configuration ## Apache In both dev and prod environments we have Node configured to run on port 3000, with Apache proxying it across to port 80. The reason for Apache is so that we can use other tech stacks alongside Node. In particular, we’ve been using a MySQL administration web frontend called PHPMyAdmin. macOS comes with an Apache installation built in. We just need to configure it a little bit. First, we’ll edit the main Apache configuration file. ``` sudo vim /private/etc/apache2/httpd.conf ``` The following individual lines each need to be uncommented: ``` LoadModule proxy_module libexec/apache2/mod_proxy.so LoadModule proxy_http_module libexec/apache2/mod_proxy_http.so LoadModule proxy_wstunnel_module libexec/apache2/mod_proxy_wstunnel.so LoadModule userdir_module libexec/apache2/mod_userdir.so LoadModule php7_module libexec/apache2/libphp7.so Include /private/etc/apache2/extra/httpd-userdir.conf ``` Next, we’ll edit the `http-userdir.conf` file. ``` sudo vim /private/etc/apache2/extra/httpd-userdir.conf ``` The following line needs to be uncommented: ``` Include /private/etc/apache2/users/*.conf ``` Now for the main course. We need to set up a configuration file for the current user. ``` sudo vim /private/etc/apache2/users/$USER.conf ``` ``` AllowOverride All Options Indexes FollowSymLinks Require all granted ProxyRequests on ProxyPass /comm/ws ws://localhost:3000/ws ProxyPass /comm/ http://localhost:3000/ RequestHeader set "X-Forwarded-Proto" expr=%{REQUEST_SCHEME} RequestHeader set "X-Forwarded-SSL" expr=%{HTTPS} ``` Make sure to replace “ashoat” on the first line above with your macOS user. You’ll also want to make sure that Apache can read your new file. ``` sudo chmod 644 /private/etc/apache2/users/$USER.conf ``` Finally, let’s restart Apache so it picks up the changes. ``` sudo apachectl restart ``` ## MySQL Next we’ll set up a MySQL user and a fresh database. We’ll start by opening up a MySQL console. ``` mysql -u root -p ``` Type in the MySQL root password you set up previously when prompted. Then, we’ll go ahead and create an empty database. ``` CREATE DATABASE comm; ``` Now we need to create a user that can access this database. For the following command, replace “password” with a unique password. ``` CREATE USER comm@localhost IDENTIFIED BY 'password'; ``` Finally, we will give permissions to this user to access this database. ``` GRANT ALL ON comm.* TO comm@localhost; ``` You can now exit the MySQL console using Ctrl+D. ## PHPMyAdmin Next we’ll set up PHPMyAdmin. If you’re familiar with an alternative MySQL administration frontend you can feel free to use it instead. Start by downloading the latest PHPMyAdmin release from their [website](https://www.phpmyadmin.net/). In the following steps, make sure to replace the version number with the one you downloaded. We’ll begin by unzipping the download. ``` unzip phpMyAdmin-5.0.1-all-languages.zip ``` In our Apache configuration above, we set Apache to handle everything in `~/Sites`, so we’ll go ahead and move PHPMyAdmin over there. ``` mkdir -p ~/Sites cd ~/Downloads mv phpMyAdmin-5.0.1-all-languages ~/Sites/phpmyadmin ``` We also need to create a “TempDir” for PHPMyAdmin. ``` pushd ~/Sites/phpmyadmin mkdir tmp chmod 777 tmp popd ``` Next we’ll go through the PHPMyAdmin setup UI. Navigate to http://localhost/~youruser/phpmyadmin/setup/, making sure to replace “youruser” with your macOS username. If there are any servers listed, go ahead and delete them. Next, select the option to add a new server. Fill out the “verbose name” under “Basic Settings”, and then make sure to set “127.0.0.1” as the hostname. For some reason, “localhost” will not work. Then tab over to “Authentication”. Select the “config” option under “Authentication type”, and then enter in the MySQL user and password that you created in the previous step. Now hit “Apply”. You should now be back on the Overview page. Hit “Download” to download the new `config.inc.php` file, and then move it into the `~/Sites/phpmyadmin` folder. ``` mv config.inc.php ~/Sites/phpmyadmin/ ``` You should now be able to access PHPMyAdmin without needing to log in. Try navigating to http://localhost/~youruser/phpmyadmin/, making sure to replace “youruser” with your macOS username. Our final step will be to configure PHP to be able to handle large files, in case you find yourself needing to deal with a database backup. PHP is configured by a `php.ini` file. First, check if you have one in `/private/etc/php.ini`. If you don’t, copy over the default: ``` cp /private/etc/php.ini.default /private/etc/php.ini ``` We just need to update three settings: `memory_limit`, `post_max_size`, and `upload_max_filesize`. All should be set to `256M`. ## Android emulator In order to test the Android app on your computer you’ll need to set up an Android emulator. To do this we’ll need to open up the AVD Manager in Android Studio. AVD stands for “Android Virtual Device”. You can access the AVD Manager from the “Welcome to Android Studio” screen that pops up when you first open the application, under “Configure”. If you already have a project open, you can access it from Tools → AVD Manager. With the the AVD Manager open, select “Create Virtual Device” on the bottom row. Feel free to select any “device definition” that includes Play Store support. On the next screen you’ll be asked to select a system image. Go for the latest version of Android that has been released. That’s currently Android 10, also known as Android Q. From there you can just hit Next and then Finish. You should then be able to start your new AVD from the AVD Manager. # Git repo ## Clone from GitHub Finally! It’s time to clone the repo from GitHub. ``` git clone git@github.com:CommE2E/comm.git ``` Once you have the repo cloned, you can run this command to pull in dependencies. ``` cd comm yarn cleaninstall ``` ## MySQL The server side needs to see some config files before things can work. The first is a config file with MySQL details. ``` cd server mkdir secrets vim secrets/db_config.json ``` The DB config file should look like this: ```json { "host": "localhost", "user": "comm", "password": "password", "database": "comm" } ``` Make sure to replace the password with the one you set up for your `comm` MySQL user earlier. New let’s run a script to setup the database. Before we can run the script, we’ll have to use Babel to transpile our source files into something Node can interpret. Babel will transpile the files in `src` into a new directory called `dist`. We also use `rsync` to copy over files that don’t need transpilation. ``` yarn babel-build yarn rsync yarn script dist/scripts/create-db.js ``` ## URLs The server needs to know some info about paths in order to properly construct URLs. ``` mkdir -p server/facts vim server/facts/url.json ``` Your `url.json` file should look like this: +```json +{ + "baseRoutePath": "/" +} +``` + +Next, we'll create a file for constructing URLs for the main app. + +``` +vim server/facts/app_url.json +``` + +Your `app_url.json` file should look like this: + ```json { "baseDomain": "http://localhost", "basePath": "/comm/", - "baseRoutePath": "/", "https": false } ``` ## Phabricator The last configuration step is to set up an account on Phabricator, where we handle code review. Start by [logging in to Phabricator](https://phabricator.ashoat.com) using your GitHub account. Next, make sure you’re inside the directory containing the SquadCal Git repository, and run the following command: ``` arc install-certificate ``` This command will help you connect your Phabricator account with the local Arcanist instance, allowing you to run `arc diff` and `arc land` commands. # Development ## Flow typechecker It’s good to run the `flow` typechecker frequently to make sure you’re not introducing any type errors. Flow treats each Yarn Workspace as a separate environment, and as such runs a separate type-checking server for each. This server is started when you first run `node_modules/.bin/flow` in each of the four Yarn Workspace folders. To make sure Flow runs from the command-line, you can edit your `$PATH` environmental variable in your `~/.bash_profile` file (or desired shell configuration file) to always include `./node_modules/.bin`. ``` export PATH=$PATH:./node_modules/.bin ``` As always, make sure you reload the `~/.bash_profile` after editing it: ``` source ~/.bash_profile ``` You should now be able to run `flow` in any of the Yarn workspaces: ``` cd lib flow ``` ## Running website Open a new terminal and run: ``` cd web yarn dev ``` This will start two processes. One is `webpack-dev-server`, which will serve the JS files. `webpack-dev-server` also makes sure the website automatically hot-reloads whenever any of the source files change. The other process is `webpack --watch`, which will build the `app.build.cjs` file, as well as rebuilding it whenever any of the source files change. The `app.build.cjs` file is consumed by the Node server in order to pre-render the initial HTML from the web source (“Server-Side Rendering”). ## Running server First make sure you have `yarn dev` running in `web` first. Next, open a new terminal and run: ``` cd server yarn dev ``` You should now be able to load the website in your web browser at http://localhost/comm/. This will run three processes. The first two are to keep the `dist` folder updated whenever the `src` folder changes. They are “watch” versions of the same Babel and `rsync` commands we used to initially create the `dist` folder (before running the `create-db.js` script above). The final process is `nodemon`, which is similar to `node` except that it restarts whenever any of its source files (in the `dist` directory) changes. Note that if you run `yarn dev` in `server` right after `yarn cleaninstall`, before Webpack is given a chance to build an `app.build.cjs` file, then Node will crash when it attempts to import that file. Just run `yarn dev` (or `yarn prod`) in `web` before attempting to run `yarn dev` in `server`. ## Running iOS First, make sure that the React Native packager is open. If you haven’t already, open a new terminal and run: ``` cd native yarn dev ``` Next, open `native/ios/SquadCal.xcworkspace` in Xcode. Select a simulator and then hit the play button to build and run the project. ## Running Android First, make sure that the React Native packager is open. If you haven’t already, open a new terminal and run: ``` cd native yarn dev ``` Next, boot up an Android simulator using Android Studio’s AVD Manager. You should have a single Android simulator (or plugged-in device) running at one time. Finally, use this command to build and run the Android app: ``` cd native yarn react-native run-android ``` # Working with Phabricator ## Creating a new diff The biggest difference between GitHub’s PR workflow and Phabricator’s “diff” workflow is that Phabricator lets you create a diff from any commit, or set of commits. In contrast, GitHub can only create PRs from branches. When you have a commit ready and want to submit a diff for code review, just run `arc diff` from within the SquadCal Git repo. Arcanist will attempt to determine the “base” for your diff automatically, but by default it will take the single most recent commit. You can see what base Arcanist thinks it should use by running `arc which`. You can also explicitly specify a base by using `arc diff --base`. For instance, `arc diff --base HEAD^` will create a diff from the most recent commit, which should be the default behavior. Keep in mind that `arc diff` always diffs the base against your current working copy. Though this nominally includes any unstashed changes you might have, `arc diff`’s interactive prompts will help you exclude unrelated changes in your working copy. It’s generally easiest to keep a 1:1 correspondence between diffs and commits. If you’re working with a stack of commits, you can use Git’s interactive rebase feature (`git rebase -i`) to run `arc diff` on each commit individually. ## Updating a diff Whereas with GitHub PRs, updates are usually created by adding on more commits, in Phabricator the easiest way to update a diff is by amending the existing commit. When you run `arc diff` on a commit for the first time, it amends the commit message to include a link to the Phabricator diff. If and when you want to update that diff, just run `arc diff` again. If you’re working with a stack of diffs, and want to update an earlier diff, you can use Git’s interactive rebase feature (`git rebase -i`) to open the stack to a particular point. Then you can amend that commit and run `arc diff` before continuing the rebase. ## Committing a diff After your diff has been accepted, you should be able to land it. To land a diff just run `arc land` from within the repository. If you’re dealing with a stack, `arc land` will make sure to only land the diffs that have been accepted, and shouldn’t land any diffs that depend on other diffs that haven’t been accepted yet. Note that you need commit rights to the repository in order to run `arc land`. If you don’t have commit rights, reach out to @ashoat for assistance. ## Creating a Herald rule Once you have access to Phabricator, you may want to set up a Herald rule so that you get CC’d on any new diffs. The way to do that in Phabricator is: 1. Go to “More Applications” on the left-hand sidebar. 2. Select “Herald” from the list. 3. Press the “Create Herald Rule” button in the upper-right corner of the screen. 4. Select “Differential Revisions” from the list. 5. Select “Personal Rule” from the list. 6. Set up your new rule to match [this one](https://phabricator.ashoat.com/H2). ## Final notes When developing, I usually just pop up three terminal windows, one for `yarn dev` in each of server, web, and native. Note that it’s currently only possible to create a user account using the iOS or Android apps. The website supports logging in, but does not support account creation. Good luck, and let @ashoat know if you have any questions! diff --git a/server/src/creators/report-creator.js b/server/src/creators/report-creator.js index 095095a04..4311fcf52 100644 --- a/server/src/creators/report-creator.js +++ b/server/src/creators/report-creator.js @@ -1,232 +1,232 @@ // @flow import _isEqual from 'lodash/fp/isEqual'; import bots from 'lib/facts/bots'; import { filterRawEntryInfosByCalendarQuery, serverEntryInfosObject, } from 'lib/shared/entry-utils'; import { messageTypes } from 'lib/types/message-types'; import { type ReportCreationRequest, type ReportCreationResponse, type ThreadInconsistencyReportCreationRequest, type EntryInconsistencyReportCreationRequest, type UserInconsistencyReportCreationRequest, reportTypes, } from 'lib/types/report-types'; import { values } from 'lib/utils/objects'; import { sanitizeAction, sanitizeState } from 'lib/utils/sanitization'; -import urlFacts from '../../facts/url'; import { dbQuery, SQL } from '../database/database'; import { fetchUsername } from '../fetchers/user-fetchers'; import { handleAsyncPromise } from '../responders/handlers'; import { createBotViewer } from '../session/bots'; import type { Viewer } from '../session/viewer'; +import { getAppURLFacts } from '../utils/urls'; import createIDs from './id-creator'; import createMessages from './message-creator'; -const { baseDomain, basePath } = urlFacts; +const { baseDomain, basePath } = getAppURLFacts(); const { squadbot } = bots; async function createReport( viewer: Viewer, request: ReportCreationRequest, ): Promise { const shouldIgnore = await ignoreReport(viewer, request); if (shouldIgnore) { return null; } const [id] = await createIDs('reports', 1); let type, report, time; if (request.type === reportTypes.THREAD_INCONSISTENCY) { ({ type, time, ...report } = request); time = time ? time : Date.now(); } else if (request.type === reportTypes.ENTRY_INCONSISTENCY) { ({ type, time, ...report } = request); } else if (request.type === reportTypes.MEDIA_MISSION) { ({ type, time, ...report } = request); } else if (request.type === reportTypes.USER_INCONSISTENCY) { ({ type, time, ...report } = request); } else { ({ type, ...report } = request); time = Date.now(); report = { ...report, preloadedState: sanitizeState(report.preloadedState), currentState: sanitizeState(report.currentState), actions: report.actions.map(sanitizeAction), }; } const row = [ id, viewer.id, type, request.platformDetails.platform, JSON.stringify(report), time, ]; const query = SQL` INSERT INTO reports (id, user, type, platform, report, creation_time) VALUES ${[row]} `; await dbQuery(query); handleAsyncPromise(sendSquadbotMessage(viewer, request, id)); return { id }; } async function sendSquadbotMessage( viewer: Viewer, request: ReportCreationRequest, reportID: string, ): Promise { const canGenerateMessage = getSquadbotMessage(request, reportID, null); if (!canGenerateMessage) { return; } const username = await fetchUsername(viewer.id); const message = getSquadbotMessage(request, reportID, username); if (!message) { return; } const time = Date.now(); await createMessages(createBotViewer(squadbot.userID), [ { type: messageTypes.TEXT, threadID: squadbot.staffThreadID, creatorID: squadbot.userID, time, text: message, }, ]); } async function ignoreReport( viewer: Viewer, request: ReportCreationRequest, ): Promise { // The below logic is to avoid duplicate inconsistency reports if ( request.type !== reportTypes.THREAD_INCONSISTENCY && request.type !== reportTypes.ENTRY_INCONSISTENCY ) { return false; } const { type, platformDetails, time } = request; if (!time) { return false; } const { platform } = platformDetails; const query = SQL` SELECT id FROM reports WHERE user = ${viewer.id} AND type = ${type} AND platform = ${platform} AND creation_time = ${time} `; const [result] = await dbQuery(query); return result.length !== 0; } function getSquadbotMessage( request: ReportCreationRequest, reportID: string, username: ?string, ): ?string { const name = username ? username : '[null]'; const { platformDetails } = request; const { platform, codeVersion } = platformDetails; const platformString = codeVersion ? `${platform} v${codeVersion}` : platform; if (request.type === reportTypes.ERROR) { return ( `${name} got an error :(\n` + `using ${platformString}\n` + `${baseDomain}${basePath}download_error_report/${reportID}` ); } else if (request.type === reportTypes.THREAD_INCONSISTENCY) { const nonMatchingThreadIDs = getInconsistentThreadIDsFromReport(request); const nonMatchingString = [...nonMatchingThreadIDs].join(', '); return ( `system detected inconsistency for ${name}!\n` + `using ${platformString}\n` + `occurred during ${request.action.type}\n` + `thread IDs that are inconsistent: ${nonMatchingString}` ); } else if (request.type === reportTypes.ENTRY_INCONSISTENCY) { const nonMatchingEntryIDs = getInconsistentEntryIDsFromReport(request); const nonMatchingString = [...nonMatchingEntryIDs].join(', '); return ( `system detected inconsistency for ${name}!\n` + `using ${platformString}\n` + `occurred during ${request.action.type}\n` + `entry IDs that are inconsistent: ${nonMatchingString}` ); } else if (request.type === reportTypes.USER_INCONSISTENCY) { const nonMatchingUserIDs = getInconsistentUserIDsFromReport(request); const nonMatchingString = [...nonMatchingUserIDs].join(', '); return ( `system detected inconsistency for ${name}!\n` + `using ${platformString}\n` + `occurred during ${request.action.type}\n` + `user IDs that are inconsistent: ${nonMatchingString}` ); } else if (request.type === reportTypes.MEDIA_MISSION) { const mediaMissionJSON = JSON.stringify(request.mediaMission); const success = request.mediaMission.result.success ? 'media mission success!' : 'media mission failed :('; return `${name} ${success}\n` + mediaMissionJSON; } else { return null; } } function findInconsistentObjectKeys( first: { +[id: string]: O }, second: { +[id: string]: O }, ): Set { const nonMatchingIDs = new Set(); for (const id in first) { if (!_isEqual(first[id])(second[id])) { nonMatchingIDs.add(id); } } for (const id in second) { if (!first[id]) { nonMatchingIDs.add(id); } } return nonMatchingIDs; } function getInconsistentThreadIDsFromReport( request: ThreadInconsistencyReportCreationRequest, ): Set { const { pushResult, beforeAction } = request; return findInconsistentObjectKeys(beforeAction, pushResult); } function getInconsistentEntryIDsFromReport( request: EntryInconsistencyReportCreationRequest, ): Set { const { pushResult, beforeAction, calendarQuery } = request; const filteredBeforeAction = filterRawEntryInfosByCalendarQuery( serverEntryInfosObject(values(beforeAction)), calendarQuery, ); const filteredAfterAction = filterRawEntryInfosByCalendarQuery( serverEntryInfosObject(values(pushResult)), calendarQuery, ); return findInconsistentObjectKeys(filteredBeforeAction, filteredAfterAction); } function getInconsistentUserIDsFromReport( request: UserInconsistencyReportCreationRequest, ): Set { const { beforeStateCheck, afterStateCheck } = request; return findInconsistentObjectKeys(beforeStateCheck, afterStateCheck); } export default createReport; diff --git a/server/src/emails/reset-password.js b/server/src/emails/reset-password.js index ac98223fb..91e46946e 100644 --- a/server/src/emails/reset-password.js +++ b/server/src/emails/reset-password.js @@ -1,50 +1,50 @@ // @flow import React from 'react'; import { Item, Span, A, renderEmail } from 'react-html-email'; import { verifyField } from 'lib/types/verify-types'; -import urlFacts from '../../facts/url'; import { createVerificationCode } from '../models/verification'; +import { getAppURLFacts } from '../utils/urls'; import sendmail from './sendmail'; import Template from './template.react'; -const { baseDomain, basePath } = urlFacts; +const { baseDomain, basePath } = getAppURLFacts(); async function sendPasswordResetEmail( userID: string, username: string, emailAddress: string, ): Promise { const code = await createVerificationCode(userID, verifyField.RESET_PASSWORD); const link = baseDomain + basePath + `verify/${code}/`; const title = 'Reset password for SquadCal'; const text = 'We received a request to reset the password associated with your ' + `account ${username} on SquadCal. If you did not issue this request, you ` + 'do not need to do anything, and your password will remain the same. ' + 'However, if you did issue this request, please visit this link to reset ' + 'your password: '; const email = ( ); const html = renderEmail(email); await sendmail.sendMail({ from: 'no-reply@squadcal.org', to: emailAddress, subject: title, html, }); } export { sendPasswordResetEmail }; diff --git a/server/src/emails/verification.js b/server/src/emails/verification.js index 0d0d3655b..9d4f80c69 100644 --- a/server/src/emails/verification.js +++ b/server/src/emails/verification.js @@ -1,57 +1,57 @@ // @flow import React from 'react'; import { Item, Span, A, renderEmail } from 'react-html-email'; import { verifyField } from 'lib/types/verify-types'; -import urlFacts from '../../facts/url'; import { createVerificationCode } from '../models/verification'; +import { getAppURLFacts } from '../utils/urls'; import sendmail from './sendmail'; import Template from './template.react'; -const { baseDomain, basePath } = urlFacts; +const { baseDomain, basePath } = getAppURLFacts(); async function sendEmailAddressVerificationEmail( userID: string, username: string, emailAddress: string, welcome: boolean = false, ): Promise { const code = await createVerificationCode(userID, verifyField.EMAIL); const link = baseDomain + basePath + `verify/${code}/`; let welcomeText = null; let action = 'verify your email'; if (welcome) { welcomeText = ( {`Welcome to SquadCal, ${username}! `} ); action = `complete your registration and ${action}`; } const title = 'Verify email for SquadCal'; const email = ( ); const html = renderEmail(email); await sendmail.sendMail({ from: 'no-reply@squadcal.org', to: emailAddress, subject: title, html, }); } export { sendEmailAddressVerificationEmail }; diff --git a/server/src/fetchers/upload-fetchers.js b/server/src/fetchers/upload-fetchers.js index 8a67db834..44568e7b4 100644 --- a/server/src/fetchers/upload-fetchers.js +++ b/server/src/fetchers/upload-fetchers.js @@ -1,121 +1,121 @@ // @flow import type { Media } from 'lib/types/media-types'; import { ServerError } from 'lib/utils/errors'; -import urlFacts from '../../facts/url'; import { dbQuery, SQL } from '../database/database'; import type { Viewer } from '../session/viewer'; +import { getAppURLFacts } from '../utils/urls'; -const { baseDomain, basePath } = urlFacts; +const { baseDomain, basePath } = getAppURLFacts(); type UploadInfo = {| content: Buffer, mime: string, |}; async function fetchUpload( viewer: Viewer, id: string, secret: string, ): Promise { const query = SQL` SELECT content, mime FROM uploads WHERE id = ${id} AND secret = ${secret} `; const [result] = await dbQuery(query); if (result.length === 0) { throw new ServerError('invalid_parameters'); } const [row] = result; const { content, mime } = row; return { content, mime }; } async function fetchUploadChunk( id: string, secret: string, pos: number, len: number, ): Promise { // We use pos + 1 because SQL is 1-indexed whereas js is 0-indexed const query = SQL` SELECT SUBSTRING(content, ${pos + 1}, ${len}) AS content, mime FROM uploads WHERE id = ${id} AND secret = ${secret} `; const [result] = await dbQuery(query); if (result.length === 0) { throw new ServerError('invalid_parameters'); } const [row] = result; const { content, mime } = row; return { content, mime, }; } // Returns total size in bytes. async function getUploadSize(id: string, secret: string): Promise { const query = SQL` SELECT LENGTH(content) AS length FROM uploads WHERE id = ${id} AND secret = ${secret} `; const [result] = await dbQuery(query); if (result.length === 0) { throw new ServerError('invalid_parameters'); } const [row] = result; const { length } = row; return length; } function getUploadURL(id: string, secret: string) { return `${baseDomain}${basePath}upload/${id}/${secret}`; } function mediaFromRow(row: Object): Media { const { uploadType: type, uploadSecret: secret } = row; const { width, height, loop } = row.uploadExtra; const id = row.uploadID.toString(); const dimensions = { width, height }; const uri = getUploadURL(id, secret); if (type === 'photo') { return { id, type: 'photo', uri, dimensions }; } else if (loop) { // $FlowFixMe add thumbnailID, thumbnailURI once they're in DB return { id, type: 'video', uri, dimensions, loop }; } else { // $FlowFixMe add thumbnailID, thumbnailURI once they're in DB return { id, type: 'video', uri, dimensions }; } } async function fetchMedia( viewer: Viewer, mediaIDs: $ReadOnlyArray, ): Promise<$ReadOnlyArray> { const query = SQL` SELECT id AS uploadID, secret AS uploadSecret, type AS uploadType, extra AS uploadExtra FROM uploads WHERE id IN (${mediaIDs}) AND uploader = ${viewer.id} AND container IS NULL `; const [result] = await dbQuery(query); return result.map(mediaFromRow); } export { fetchUpload, fetchUploadChunk, getUploadSize, getUploadURL, mediaFromRow, fetchMedia, }; diff --git a/server/src/responders/website-responders.js b/server/src/responders/website-responders.js index f1346f97d..5397603bb 100644 --- a/server/src/responders/website-responders.js +++ b/server/src/responders/website-responders.js @@ -1,331 +1,331 @@ // @flow import html from 'common-tags/lib/html'; import type { $Response, $Request } from 'express'; import fs from 'fs'; import _keyBy from 'lodash/fp/keyBy'; import React from 'react'; import ReactDOMServer from 'react-dom/server'; import { Provider } from 'react-redux'; import { Route, StaticRouter } from 'react-router'; import { createStore, type Store } from 'redux'; import { promisify } from 'util'; import { daysToEntriesFromEntryInfos } from 'lib/reducers/entry-reducer'; import { freshMessageStore } from 'lib/reducers/message-reducer'; import { mostRecentReadThread } from 'lib/selectors/thread-selectors'; import { mostRecentMessageTimestamp } from 'lib/shared/message-utils'; import { threadHasPermission } from 'lib/shared/thread-utils'; import { defaultCalendarFilters } from 'lib/types/filter-types'; import { defaultNumberPerThread } from 'lib/types/message-types'; import { defaultConnectionInfo } from 'lib/types/socket-types'; import { threadPermissions } from 'lib/types/thread-types'; import type { ServerVerificationResult } from 'lib/types/verify-types'; import { currentDateInTimeZone } from 'lib/utils/date-utils'; import { ServerError } from 'lib/utils/errors'; import { promiseAll } from 'lib/utils/promises'; import App from 'web/dist/app.build.cjs'; import { reducer } from 'web/redux/redux-setup'; import type { AppState, Action } from 'web/redux/redux-setup'; import getTitle from 'web/title/getTitle'; import { navInfoFromURL } from 'web/url-utils'; -import urlFacts from '../../facts/url'; import { fetchEntryInfos } from '../fetchers/entry-fetchers'; import { fetchMessageInfos } from '../fetchers/message-fetchers'; import { fetchThreadInfos } from '../fetchers/thread-fetchers'; import { fetchCurrentUserInfo, fetchKnownUserInfos, } from '../fetchers/user-fetchers'; import { handleCodeVerificationRequest } from '../models/verification'; import { setNewSession } from '../session/cookies'; import { Viewer } from '../session/viewer'; import { streamJSON, waitForStream } from '../utils/json-stream'; +import { getAppURLFacts } from '../utils/urls'; -const { basePath, baseDomain } = urlFacts; +const { basePath, baseDomain } = getAppURLFacts(); const { renderToNodeStream } = ReactDOMServer; const baseURL = basePath.replace(/\/$/, ''); const baseHref = baseDomain + baseURL; const access = promisify(fs.access); const googleFontsURL = 'https://fonts.googleapis.com/css?family=Open+Sans:300,600%7CAnaheim'; const localFontsURL = 'fonts/local-fonts.css'; async function getFontsURL() { try { await access(localFontsURL); return localFontsURL; } catch { return googleFontsURL; } } type AssetInfo = {| jsURL: string, fontsURL: string, cssInclude: string |}; let assetInfo: ?AssetInfo = null; async function getAssetInfo() { if (assetInfo) { return assetInfo; } if (process.env.NODE_ENV === 'dev') { const fontsURL = await getFontsURL(); assetInfo = { jsURL: 'http://localhost:8080/dev.build.js', fontsURL, cssInclude: '', }; return assetInfo; } // $FlowFixMe compiled/assets.json doesn't always exist const { default: assets } = await import('../../compiled/assets'); assetInfo = { jsURL: `compiled/${assets.browser.js}`, fontsURL: googleFontsURL, cssInclude: html` `, }; return assetInfo; } async function websiteResponder( viewer: Viewer, req: $Request, res: $Response, ): Promise { let initialNavInfo; try { initialNavInfo = navInfoFromURL(req.url, { now: currentDateInTimeZone(viewer.timeZone), }); } catch (e) { throw new ServerError(e.message); } const calendarQuery = { startDate: initialNavInfo.startDate, endDate: initialNavInfo.endDate, filters: defaultCalendarFilters, }; const threadSelectionCriteria = { joinedThreads: true }; const initialTime = Date.now(); const assetInfoPromise = getAssetInfo(); const threadInfoPromise = fetchThreadInfos(viewer); const messageInfoPromise = fetchMessageInfos( viewer, threadSelectionCriteria, defaultNumberPerThread, ); const entryInfoPromise = fetchEntryInfos(viewer, [calendarQuery]); const currentUserInfoPromise = fetchCurrentUserInfo(viewer); const serverVerificationResultPromise = handleVerificationRequest( viewer, initialNavInfo.verify, ); const userInfoPromise = fetchKnownUserInfos(viewer); const sessionIDPromise = (async () => { if (viewer.loggedIn) { await setNewSession(viewer, calendarQuery, initialTime); } return viewer.sessionID; })(); const threadStorePromise = (async () => { const { threadInfos } = await threadInfoPromise; return { threadInfos, inconsistencyReports: [] }; })(); const messageStorePromise = (async () => { const [ { threadInfos }, { rawMessageInfos, truncationStatuses }, ] = await Promise.all([threadInfoPromise, messageInfoPromise]); return freshMessageStore( rawMessageInfos, truncationStatuses, mostRecentMessageTimestamp(rawMessageInfos, initialTime), threadInfos, ); })(); const entryStorePromise = (async () => { const { rawEntryInfos } = await entryInfoPromise; return { entryInfos: _keyBy('id')(rawEntryInfos), daysToEntries: daysToEntriesFromEntryInfos(rawEntryInfos), lastUserInteractionCalendar: initialTime, inconsistencyReports: [], }; })(); const userStorePromise = (async () => { const userInfos = await userInfoPromise; return { userInfos, inconsistencyReports: [] }; })(); const navInfoPromise = (async () => { const [{ threadInfos }, messageStore] = await Promise.all([ threadInfoPromise, messageStorePromise, ]); const finalNavInfo = initialNavInfo; const requestedActiveChatThreadID = finalNavInfo.activeChatThreadID; if ( requestedActiveChatThreadID && !threadHasPermission( threadInfos[requestedActiveChatThreadID], threadPermissions.VISIBLE, ) ) { finalNavInfo.activeChatThreadID = null; } if (!finalNavInfo.activeChatThreadID) { const mostRecentThread = mostRecentReadThread(messageStore, threadInfos); if (mostRecentThread) { finalNavInfo.activeChatThreadID = mostRecentThread; } } return finalNavInfo; })(); const { jsURL, fontsURL, cssInclude } = await assetInfoPromise; // prettier-ignore res.write(html` ${getTitle(0)} ${cssInclude}
`); const statePromises = { navInfo: navInfoPromise, currentUserInfo: currentUserInfoPromise, sessionID: sessionIDPromise, serverVerificationResult: serverVerificationResultPromise, entryStore: entryStorePromise, threadStore: threadStorePromise, userStore: userStorePromise, messageStore: messageStorePromise, updatesCurrentAsOf: initialTime, loadingStatuses: {}, calendarFilters: defaultCalendarFilters, // We can use paths local to the on web urlPrefix: '', windowDimensions: { width: 0, height: 0 }, baseHref, connection: { ...defaultConnectionInfo('web', viewer.timeZone), actualizedCalendarQuery: calendarQuery, }, watchedThreadIDs: [], lifecycleState: 'active', nextLocalID: 0, queuedReports: [], timeZone: viewer.timeZone, userAgent: viewer.userAgent, cookie: undefined, deviceToken: undefined, dataLoaded: viewer.loggedIn, windowActive: true, }; const stateResult = await promiseAll(statePromises); const state: AppState = { ...stateResult }; const store: Store = createStore(reducer, state); const routerContext = {}; const reactStream = renderToNodeStream( , ); if (routerContext.url) { throw new ServerError('URL modified during server render!'); } reactStream.pipe(res, { end: false }); await waitForStream(reactStream); res.write(html`
`); } async function handleVerificationRequest( viewer: Viewer, code: ?string, ): Promise { if (!code) { return null; } try { return await handleCodeVerificationRequest(viewer, code); } catch (e) { if (e instanceof ServerError && e.message === 'invalid_code') { return { success: false }; } throw e; } } export { websiteResponder }; diff --git a/server/src/server.js b/server/src/server.js index fe5acbe58..1a4afa414 100644 --- a/server/src/server.js +++ b/server/src/server.js @@ -1,95 +1,95 @@ // @flow import cluster from 'cluster'; import cookieParser from 'cookie-parser'; import express from 'express'; import expressWs from 'express-ws'; import os from 'os'; -import urlFacts from '../facts/url'; import './cron/cron'; import { jsonEndpoints } from './endpoints'; import { jsonHandler, downloadHandler, htmlHandler, uploadHandler, } from './responders/handlers'; import { errorReportDownloadResponder } from './responders/report-responders'; import { websiteResponder } from './responders/website-responders'; import { onConnection } from './socket/socket'; import { multerProcessor, multimediaUploadResponder, uploadDownloadResponder, } from './uploads/uploads'; +import { getGlobalURLFacts } from './utils/urls'; -const { baseRoutePath } = urlFacts; +const { baseRoutePath } = getGlobalURLFacts(); if (cluster.isMaster) { const cpuCount = os.cpus().length; for (let i = 0; i < cpuCount; i++) { cluster.fork(); } cluster.on('exit', () => cluster.fork()); } else { const server = express(); expressWs(server); server.use(express.json({ limit: '50mb' })); server.use(cookieParser()); const router = express.Router(); router.use('/images', express.static('images')); if (process.env.NODE_ENV === 'dev') { router.use('/fonts', express.static('fonts')); } router.use('/misc', express.static('misc')); router.use( '/.well-known', express.static( '.well-known', // Necessary for apple-app-site-association file { setHeaders: (res) => res.setHeader('Content-Type', 'application/json'), }, ), ); const compiledFolderOptions = process.env.NODE_ENV === 'dev' ? undefined : { maxAge: '1y', immutable: true }; router.use('/compiled', express.static('compiled', compiledFolderOptions)); router.use('/', express.static('icons')); for (const endpoint in jsonEndpoints) { // $FlowFixMe Flow thinks endpoint is string const responder = jsonEndpoints[endpoint]; const expectCookieInvalidation = endpoint === 'log_out'; router.post( `/${endpoint}`, jsonHandler(responder, expectCookieInvalidation), ); } router.get( '/download_error_report/:reportID', downloadHandler(errorReportDownloadResponder), ); router.get( '/upload/:uploadID/:secret', downloadHandler(uploadDownloadResponder), ); // $FlowFixMe express-ws has side effects that can't be typed router.ws('/ws', onConnection); router.get('*', htmlHandler(websiteResponder)); router.post( '/upload_multimedia', multerProcessor, uploadHandler(multimediaUploadResponder), ); server.use(baseRoutePath, router); server.listen(parseInt(process.env.PORT, 10) || 3000, 'localhost'); } diff --git a/server/src/session/cookies.js b/server/src/session/cookies.js index 09062e86e..564cf65df 100644 --- a/server/src/session/cookies.js +++ b/server/src/session/cookies.js @@ -1,807 +1,807 @@ // @flow import crypto from 'crypto'; import type { $Response, $Request } from 'express'; import invariant from 'invariant'; import bcrypt from 'twin-bcrypt'; import url from 'url'; import { hasMinCodeVersion } from 'lib/shared/version-utils'; import type { Shape } from 'lib/types/core'; import type { Platform, PlatformDetails } from 'lib/types/device-types'; import type { CalendarQuery } from 'lib/types/entry-types'; import { type ServerSessionChange, cookieLifetime, cookieSources, type CookieSource, cookieTypes, sessionIdentifierTypes, type SessionIdentifierType, } from 'lib/types/session-types'; import type { InitialClientSocketMessage } from 'lib/types/socket-types'; import type { UserInfo } from 'lib/types/user-types'; import { values } from 'lib/utils/objects'; import { promiseAll } from 'lib/utils/promises'; -import urlFacts from '../../facts/url'; import createIDs from '../creators/id-creator'; import { createSession } from '../creators/session-creator'; import { dbQuery, SQL } from '../database/database'; import { deleteCookie } from '../deleters/cookie-deleters'; import { handleAsyncPromise } from '../responders/handlers'; import { clearDeviceToken } from '../updaters/device-token-updaters'; import { updateThreadMembers } from '../updaters/thread-updaters'; import { assertSecureRequest } from '../utils/security-utils'; +import { getAppURLFacts } from '../utils/urls'; import { Viewer } from './viewer'; import type { AnonymousViewerData, UserViewerData } from './viewer'; -const { baseDomain, basePath, https } = urlFacts; +const { baseDomain, basePath, https } = getAppURLFacts(); function cookieIsExpired(lastUsed: number) { return lastUsed + cookieLifetime <= Date.now(); } type SessionParameterInfo = {| isSocket: boolean, sessionID: ?string, sessionIdentifierType: SessionIdentifierType, ipAddress: string, userAgent: ?string, |}; type FetchViewerResult = | {| type: 'valid', viewer: Viewer |} | InvalidFetchViewerResult; type InvalidFetchViewerResult = | {| type: 'nonexistant', cookieName: ?string, cookieSource: ?CookieSource, sessionParameterInfo: SessionParameterInfo, |} | {| type: 'invalidated', cookieName: string, cookieID: string, cookieSource: CookieSource, sessionParameterInfo: SessionParameterInfo, platformDetails: ?PlatformDetails, deviceToken: ?string, |}; async function fetchUserViewer( cookie: string, cookieSource: CookieSource, sessionParameterInfo: SessionParameterInfo, ): Promise { const [cookieID, cookiePassword] = cookie.split(':'); if (!cookieID || !cookiePassword) { return { type: 'nonexistant', cookieName: cookieTypes.USER, cookieSource, sessionParameterInfo, }; } const query = SQL` SELECT hash, user, last_used, platform, device_token, versions FROM cookies WHERE id = ${cookieID} AND user IS NOT NULL `; const [[result], allSessionInfo] = await Promise.all([ dbQuery(query), fetchSessionInfo(sessionParameterInfo, cookieID), ]); if (result.length === 0) { return { type: 'nonexistant', cookieName: cookieTypes.USER, cookieSource, sessionParameterInfo, }; } let sessionID = null, sessionInfo = null; if (allSessionInfo) { ({ sessionID, ...sessionInfo } = allSessionInfo); } const cookieRow = result[0]; let platformDetails = null; if (cookieRow.versions) { platformDetails = { platform: cookieRow.platform, codeVersion: cookieRow.versions.codeVersion, stateVersion: cookieRow.versions.stateVersion, }; } else { platformDetails = { platform: cookieRow.platform }; } const deviceToken = cookieRow.device_token; if ( !bcrypt.compareSync(cookiePassword, cookieRow.hash) || cookieIsExpired(cookieRow.last_used) ) { return { type: 'invalidated', cookieName: cookieTypes.USER, cookieID, cookieSource, sessionParameterInfo, platformDetails, deviceToken, }; } const userID = cookieRow.user.toString(); const viewer = new Viewer({ isSocket: sessionParameterInfo.isSocket, loggedIn: true, id: userID, platformDetails, deviceToken, userID, cookieSource, cookieID, cookiePassword, sessionIdentifierType: sessionParameterInfo.sessionIdentifierType, sessionID, sessionInfo, isScriptViewer: false, ipAddress: sessionParameterInfo.ipAddress, userAgent: sessionParameterInfo.userAgent, }); return { type: 'valid', viewer }; } async function fetchAnonymousViewer( cookie: string, cookieSource: CookieSource, sessionParameterInfo: SessionParameterInfo, ): Promise { const [cookieID, cookiePassword] = cookie.split(':'); if (!cookieID || !cookiePassword) { return { type: 'nonexistant', cookieName: cookieTypes.ANONYMOUS, cookieSource, sessionParameterInfo, }; } const query = SQL` SELECT last_used, hash, platform, device_token, versions FROM cookies WHERE id = ${cookieID} AND user IS NULL `; const [[result], allSessionInfo] = await Promise.all([ dbQuery(query), fetchSessionInfo(sessionParameterInfo, cookieID), ]); if (result.length === 0) { return { type: 'nonexistant', cookieName: cookieTypes.ANONYMOUS, cookieSource, sessionParameterInfo, }; } let sessionID = null, sessionInfo = null; if (allSessionInfo) { ({ sessionID, ...sessionInfo } = allSessionInfo); } const cookieRow = result[0]; let platformDetails = null; if (cookieRow.platform && cookieRow.versions) { platformDetails = { platform: cookieRow.platform, codeVersion: cookieRow.versions.codeVersion, stateVersion: cookieRow.versions.stateVersion, }; } else if (cookieRow.platform) { platformDetails = { platform: cookieRow.platform }; } const deviceToken = cookieRow.device_token; if ( !bcrypt.compareSync(cookiePassword, cookieRow.hash) || cookieIsExpired(cookieRow.last_used) ) { return { type: 'invalidated', cookieName: cookieTypes.ANONYMOUS, cookieID, cookieSource, sessionParameterInfo, platformDetails, deviceToken, }; } const viewer = new Viewer({ isSocket: sessionParameterInfo.isSocket, loggedIn: false, id: cookieID, platformDetails, deviceToken, cookieSource, cookieID, cookiePassword, sessionIdentifierType: sessionParameterInfo.sessionIdentifierType, sessionID, sessionInfo, isScriptViewer: false, ipAddress: sessionParameterInfo.ipAddress, userAgent: sessionParameterInfo.userAgent, }); return { type: 'valid', viewer }; } type SessionInfo = {| sessionID: ?string, lastValidated: number, calendarQuery: CalendarQuery, |}; async function fetchSessionInfo( sessionParameterInfo: SessionParameterInfo, cookieID: string, ): Promise { const { sessionID } = sessionParameterInfo; const session = sessionID !== undefined ? sessionID : cookieID; if (!session) { return null; } const query = SQL` SELECT query, last_validated FROM sessions WHERE id = ${session} AND cookie = ${cookieID} `; const [result] = await dbQuery(query); if (result.length === 0) { return null; } return { sessionID, lastValidated: result[0].last_validated, calendarQuery: result[0].query, }; } // This function is meant to consume a cookie that has already been processed. // That means it doesn't have any logic to handle an invalid cookie, and it // doesn't update the cookie's last_used timestamp. async function fetchViewerFromCookieData( req: $Request, sessionParameterInfo: SessionParameterInfo, ): Promise { let viewerResult; const { user, anonymous } = req.cookies; if (user) { viewerResult = await fetchUserViewer( user, cookieSources.HEADER, sessionParameterInfo, ); } else if (anonymous) { viewerResult = await fetchAnonymousViewer( anonymous, cookieSources.HEADER, sessionParameterInfo, ); } else { return { type: 'nonexistant', cookieName: null, cookieSource: null, sessionParameterInfo, }; } // We protect against CSRF attacks by making sure that on web, // non-GET requests cannot use a bare cookie for session identification if (viewerResult.type === 'valid') { const { viewer } = viewerResult; invariant( req.method === 'GET' || viewer.sessionIdentifierType !== sessionIdentifierTypes.COOKIE_ID || viewer.platform !== 'web', 'non-GET request from web using sessionIdentifierTypes.COOKIE_ID', ); } return viewerResult; } async function fetchViewerFromRequestBody( body: mixed, sessionParameterInfo: SessionParameterInfo, ): Promise { if (!body || typeof body !== 'object') { return { type: 'nonexistant', cookieName: null, cookieSource: null, sessionParameterInfo, }; } const cookiePair = body.cookie; if (cookiePair === null || cookiePair === '') { return { type: 'nonexistant', cookieName: null, cookieSource: cookieSources.BODY, sessionParameterInfo, }; } if (!cookiePair || typeof cookiePair !== 'string') { return { type: 'nonexistant', cookieName: null, cookieSource: null, sessionParameterInfo, }; } const [type, cookie] = cookiePair.split('='); if (type === cookieTypes.USER && cookie) { return await fetchUserViewer( cookie, cookieSources.BODY, sessionParameterInfo, ); } else if (type === cookieTypes.ANONYMOUS && cookie) { return await fetchAnonymousViewer( cookie, cookieSources.BODY, sessionParameterInfo, ); } return { type: 'nonexistant', cookieName: null, cookieSource: null, sessionParameterInfo, }; } function getSessionParameterInfoFromRequestBody( req: $Request, ): SessionParameterInfo { const body = (req.body: any); let sessionID = body.sessionID !== undefined || req.method !== 'GET' ? body.sessionID : null; if (sessionID === '') { sessionID = null; } const sessionIdentifierType = req.method === 'GET' || sessionID !== undefined ? sessionIdentifierTypes.BODY_SESSION_ID : sessionIdentifierTypes.COOKIE_ID; const ipAddress = req.get('X-Forwarded-For'); invariant(ipAddress, 'X-Forwarded-For header missing'); return { isSocket: false, sessionID, sessionIdentifierType, ipAddress, userAgent: req.get('User-Agent'), }; } async function fetchViewerForJSONRequest(req: $Request): Promise { assertSecureRequest(req); const sessionParameterInfo = getSessionParameterInfoFromRequestBody(req); let result = await fetchViewerFromRequestBody(req.body, sessionParameterInfo); if ( result.type === 'nonexistant' && (result.cookieSource === null || result.cookieSource === undefined) ) { result = await fetchViewerFromCookieData(req, sessionParameterInfo); } return await handleFetchViewerResult(result); } const webPlatformDetails = { platform: 'web' }; async function fetchViewerForHomeRequest(req: $Request): Promise { assertSecureRequest(req); const sessionParameterInfo = getSessionParameterInfoFromRequestBody(req); const result = await fetchViewerFromCookieData(req, sessionParameterInfo); return await handleFetchViewerResult(result, webPlatformDetails); } async function fetchViewerForSocket( req: $Request, clientMessage: InitialClientSocketMessage, ): Promise { assertSecureRequest(req); const { sessionIdentification } = clientMessage.payload; const { sessionID } = sessionIdentification; const ipAddress = req.get('X-Forwarded-For'); invariant(ipAddress, 'X-Forwarded-For header missing'); const sessionParameterInfo = { isSocket: true, sessionID, sessionIdentifierType: sessionID !== undefined ? sessionIdentifierTypes.BODY_SESSION_ID : sessionIdentifierTypes.COOKIE_ID, ipAddress, userAgent: req.get('User-Agent'), }; let result = await fetchViewerFromRequestBody( clientMessage.payload.sessionIdentification, sessionParameterInfo, ); if ( result.type === 'nonexistant' && (result.cookieSource === null || result.cookieSource === undefined) ) { result = await fetchViewerFromCookieData(req, sessionParameterInfo); } if (result.type === 'valid') { return result.viewer; } const promises = {}; if (result.cookieSource === cookieSources.BODY) { // We initialize a socket's Viewer after the WebSocket handshake, since to // properly initialize the Viewer we need a bunch of data, but that data // can't be sent until after the handshake. Consequently, by the time we // know that a cookie may be invalid, we are no longer communicating via // HTTP, and have no way to set a new cookie for HEADER (web) clients. const platformDetails = result.type === 'invalidated' ? result.platformDetails : null; const deviceToken = result.type === 'invalidated' ? result.deviceToken : null; promises.anonymousViewerData = createNewAnonymousCookie({ platformDetails, deviceToken, }); } if (result.type === 'invalidated') { promises.deleteCookie = deleteCookie(result.cookieID); } const { anonymousViewerData } = await promiseAll(promises); if (!anonymousViewerData) { return null; } return createViewerForInvalidFetchViewerResult(result, anonymousViewerData); } async function handleFetchViewerResult( result: FetchViewerResult, inputPlatformDetails?: PlatformDetails, ) { if (result.type === 'valid') { return result.viewer; } let platformDetails = inputPlatformDetails; if (!platformDetails && result.type === 'invalidated') { platformDetails = result.platformDetails; } const deviceToken = result.type === 'invalidated' ? result.deviceToken : null; const [anonymousViewerData] = await Promise.all([ createNewAnonymousCookie({ platformDetails, deviceToken }), result.type === 'invalidated' ? deleteCookie(result.cookieID) : null, ]); return createViewerForInvalidFetchViewerResult(result, anonymousViewerData); } function createViewerForInvalidFetchViewerResult( result: InvalidFetchViewerResult, anonymousViewerData: AnonymousViewerData, ): Viewer { // If a null cookie was specified in the request body, result.cookieSource // will still be BODY here. The only way it would be null or undefined here // is if there was no cookie specified in either the body or the header, in // which case we default to returning the new cookie in the response header. const cookieSource = result.cookieSource !== null && result.cookieSource !== undefined ? result.cookieSource : cookieSources.HEADER; const viewer = new Viewer({ ...anonymousViewerData, cookieSource, sessionIdentifierType: result.sessionParameterInfo.sessionIdentifierType, isSocket: result.sessionParameterInfo.isSocket, ipAddress: result.sessionParameterInfo.ipAddress, userAgent: result.sessionParameterInfo.userAgent, }); viewer.sessionChanged = true; // If cookieName is falsey, that tells us that there was no cookie specified // in the request, which means we can't be invalidating anything. if (result.cookieName) { viewer.cookieInvalidated = true; viewer.initialCookieName = result.cookieName; } return viewer; } const domainAsURL = new url.URL(baseDomain); function addSessionChangeInfoToResult( viewer: Viewer, res: $Response, result: Object, ) { let threadInfos = {}, userInfos = {}; if (result.cookieChange) { ({ threadInfos, userInfos } = result.cookieChange); } let sessionChange; if (viewer.cookieInvalidated) { sessionChange = ({ cookieInvalidated: true, threadInfos, userInfos: (values(userInfos).map((a) => a): UserInfo[]), currentUserInfo: { id: viewer.cookieID, anonymous: true, }, }: ServerSessionChange); } else { sessionChange = ({ cookieInvalidated: false, threadInfos, userInfos: (values(userInfos).map((a) => a): UserInfo[]), }: ServerSessionChange); } if (viewer.cookieSource === cookieSources.BODY) { sessionChange.cookie = viewer.cookiePairString; } else { addActualHTTPCookie(viewer, res); } if (viewer.sessionIdentifierType === sessionIdentifierTypes.BODY_SESSION_ID) { sessionChange.sessionID = viewer.sessionID ? viewer.sessionID : null; } result.cookieChange = sessionChange; } type AnonymousCookieCreationParams = Shape<{| +platformDetails: ?PlatformDetails, +deviceToken: ?string, |}>; const defaultPlatformDetails = {}; // The result of this function should not be passed directly to the Viewer // constructor. Instead, it should be passed to viewer.setNewCookie. There are // several fields on AnonymousViewerData that are not set by this function: // sessionIdentifierType, cookieSource, ipAddress, and userAgent. These // parameters all depend on the initial request. If the result of this function // is passed to the Viewer constructor directly, the resultant Viewer object // will throw whenever anybody attempts to access the relevant properties. async function createNewAnonymousCookie( params: AnonymousCookieCreationParams, ): Promise { const { platformDetails, deviceToken } = params; const { platform, ...versions } = platformDetails || defaultPlatformDetails; const versionsString = Object.keys(versions).length > 0 ? JSON.stringify(versions) : null; const time = Date.now(); const cookiePassword = crypto.randomBytes(32).toString('hex'); const cookieHash = bcrypt.hashSync(cookiePassword); const [[id]] = await Promise.all([ createIDs('cookies', 1), deviceToken ? clearDeviceToken(deviceToken) : undefined, ]); const cookieRow = [ id, cookieHash, null, platform, time, time, deviceToken, versionsString, ]; const query = SQL` INSERT INTO cookies(id, hash, user, platform, creation_time, last_used, device_token, versions) VALUES ${[cookieRow]} `; await dbQuery(query); return { loggedIn: false, id, platformDetails, deviceToken, cookieID: id, cookiePassword, sessionID: undefined, sessionInfo: null, cookieInsertedThisRequest: true, isScriptViewer: false, }; } type UserCookieCreationParams = {| platformDetails: PlatformDetails, deviceToken?: ?string, |}; // The result of this function should never be passed directly to the Viewer // constructor. Instead, it should be passed to viewer.setNewCookie. There are // several fields on UserViewerData that are not set by this function: // sessionID, sessionIdentifierType, cookieSource, and ipAddress. These // parameters all depend on the initial request. If the result of this function // is passed to the Viewer constructor directly, the resultant Viewer object // will throw whenever anybody attempts to access the relevant properties. async function createNewUserCookie( userID: string, params: UserCookieCreationParams, ): Promise { const { platformDetails, deviceToken } = params; const { platform, ...versions } = platformDetails || defaultPlatformDetails; const versionsString = Object.keys(versions).length > 0 ? JSON.stringify(versions) : null; const time = Date.now(); const cookiePassword = crypto.randomBytes(32).toString('hex'); const cookieHash = bcrypt.hashSync(cookiePassword); const [[cookieID]] = await Promise.all([ createIDs('cookies', 1), deviceToken ? clearDeviceToken(deviceToken) : undefined, ]); const cookieRow = [ cookieID, cookieHash, userID, platform, time, time, deviceToken, versionsString, ]; const query = SQL` INSERT INTO cookies(id, hash, user, platform, creation_time, last_used, device_token, versions) VALUES ${[cookieRow]} `; await dbQuery(query); return { loggedIn: true, id: userID, platformDetails, deviceToken, userID, cookieID, sessionID: undefined, sessionInfo: null, cookiePassword, cookieInsertedThisRequest: true, isScriptViewer: false, }; } // This gets called after createNewUserCookie and from websiteResponder. If the // Viewer's sessionIdentifierType is COOKIE_ID then the cookieID is used as the // session identifier; otherwise, a new ID is created for the session. async function setNewSession( viewer: Viewer, calendarQuery: CalendarQuery, initialLastUpdate: number, ): Promise { if (viewer.sessionIdentifierType !== sessionIdentifierTypes.COOKIE_ID) { const [sessionID] = await createIDs('sessions', 1); viewer.setSessionID(sessionID); } await createSession(viewer, calendarQuery, initialLastUpdate); } async function extendCookieLifespan(cookieID: string) { const time = Date.now(); const query = SQL` UPDATE cookies SET last_used = ${time} WHERE id = ${cookieID} `; await dbQuery(query); } function addCookieToJSONResponse( viewer: Viewer, res: $Response, result: Object, expectCookieInvalidation: boolean, ) { if (expectCookieInvalidation) { viewer.cookieInvalidated = false; } if (!viewer.getData().cookieInsertedThisRequest) { handleAsyncPromise(extendCookieLifespan(viewer.cookieID)); } if (viewer.sessionChanged) { addSessionChangeInfoToResult(viewer, res, result); } else if (viewer.cookieSource !== cookieSources.BODY) { addActualHTTPCookie(viewer, res); } } function addCookieToHomeResponse(viewer: Viewer, res: $Response) { if (!viewer.getData().cookieInsertedThisRequest) { handleAsyncPromise(extendCookieLifespan(viewer.cookieID)); } addActualHTTPCookie(viewer, res); } const cookieOptions = { domain: domainAsURL.hostname, path: basePath, httpOnly: true, secure: https, maxAge: cookieLifetime, sameSite: 'Strict', }; function addActualHTTPCookie(viewer: Viewer, res: $Response) { res.cookie(viewer.cookieName, viewer.cookieString, cookieOptions); if (viewer.cookieName !== viewer.initialCookieName) { res.clearCookie(viewer.initialCookieName, cookieOptions); } } async function setCookiePlatform( viewer: Viewer, platform: Platform, ): Promise { const newPlatformDetails = { ...viewer.platformDetails, platform }; viewer.setPlatformDetails(newPlatformDetails); const query = SQL` UPDATE cookies SET platform = ${platform} WHERE id = ${viewer.cookieID} `; await dbQuery(query); } async function setCookiePlatformDetails( viewer: Viewer, platformDetails: PlatformDetails, ): Promise { if ( hasMinCodeVersion(platformDetails, 70) && !hasMinCodeVersion(viewer.platformDetails, 70) ) { await updateThreadMembers(viewer); } viewer.setPlatformDetails(platformDetails); const { platform, ...versions } = platformDetails; const versionsString = Object.keys(versions).length > 0 ? JSON.stringify(versions) : null; const query = SQL` UPDATE cookies SET platform = ${platform}, versions = ${versionsString} WHERE id = ${viewer.cookieID} `; await dbQuery(query); } export { fetchViewerForJSONRequest, fetchViewerForHomeRequest, fetchViewerForSocket, createNewAnonymousCookie, createNewUserCookie, setNewSession, extendCookieLifespan, addCookieToJSONResponse, addCookieToHomeResponse, setCookiePlatform, setCookiePlatformDetails, }; diff --git a/server/src/utils/security-utils.js b/server/src/utils/security-utils.js index bd7bb61c6..9686c25bb 100644 --- a/server/src/utils/security-utils.js +++ b/server/src/utils/security-utils.js @@ -1,15 +1,15 @@ // @flow import type { $Request } from 'express'; -import urlFacts from '../../facts/url'; +import { getAppURLFacts } from './urls'; -const { https } = urlFacts; +const { https } = getAppURLFacts(); function assertSecureRequest(req: $Request) { if (https && req.get('X-Forwarded-SSL') !== 'on') { throw new Error('insecure request'); } } export { assertSecureRequest }; diff --git a/server/src/utils/urls.js b/server/src/utils/urls.js new file mode 100644 index 000000000..e9139ca88 --- /dev/null +++ b/server/src/utils/urls.js @@ -0,0 +1,24 @@ +// @flow + +import appURLFacts from '../../facts/app_url'; +import baseURLFacts from '../../facts/url'; + +type GlobalURLFacts = {| + +baseRoutePath: string, +|}; + +function getGlobalURLFacts(): GlobalURLFacts { + return baseURLFacts; +} + +type SiteURLFacts = {| + +baseDomain: string, + +basePath: string, + +https: boolean, +|}; + +function getAppURLFacts(): SiteURLFacts { + return appURLFacts; +} + +export { getGlobalURLFacts, getAppURLFacts };