Paths

Table of Contentst

-
services/blob/
-
blob/
-
CMakeLists.txt
-
dev/
-
DatabaseSimulator.h
-
DevTools.h
-
DevTools.cpp
-
src/
-
AwsS3Bucket.dev.cpp
-
AwsTools.cpp
-
DatabaseManager.h
-
DatabaseManager.dev.cpp
-
MultiPartUploader.dev.cpp
-
test/
-
MultiPartUploadTest.cpp

[services] Dev mode - Set up Blob service
ClosedPublic
Actions

Authored by • karol on Apr 11 2022, 6:01 AM.

Details

Reviewers

tomek
• jim

Commits

rCOMMe82deb6b61df: [services] Dev mode - Set up Blob service

Summary

Depends on D3690

Modify the code of the blob service so the dev mode that uses the local cloud can be properly launched

Test Plan

cd services
docker-compose down
yarn run-local-cloud
yarn test-blob-service-dev-mode

Diff Detail

Repository

rCOMM Comm

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

• karol created this revision.Apr 11 2022, 6:01 AM

Herald added subscribers: • abosh, • benschac, atul and 3 others. · View Herald TranscriptApr 11 2022, 6:01 AM

Harbormaster completed remote builds in B8060: Diff 11279.Apr 11 2022, 6:08 AM

• karol requested review of this revision.Apr 11 2022, 6:08 AM

• karol edited the summary of this revision. (Show Details)Apr 11 2022, 6:38 AM

• karol edited the test plan for this revision. (Show Details)

• karol added reviewers: tomek, • jim.

• karol added a parent revision: D3690: [services] Dev mode - Set up Backup service.

updating D3691: [services] Dev mode - Set up Blob service

• karol added inline comments.Apr 11 2022, 6:48 AM

services/blob/src/AwsTools.cpp
31 ↗	(On Diff #11291)	same as https://phabricator.ashoat.com/D3690#101467

Harbormaster completed remote builds in B8072: Diff 11291.Apr 11 2022, 6:52 AM

tomek requested changes to this revision.Apr 26 2022, 10:49 AM

tomek added inline comments.

services/blob/src/AwsTools.cpp
31 ↗	(On Diff #11291)	Yeah, it looks like we should avoid using compile time flag

This revision now requires changes to proceed.Apr 26 2022, 10:49 AM

I put up a follow-up for avoiding using compile-time flags D3872

To see why it comes as a follow-up, please see https://phabricator.ashoat.com/D3690#107900

• karol added a child revision: D3872: [services] Blob/Backup - Read env vars in runtime instead of compile-time flags.Apr 28 2022, 2:39 AM

In D3691#107905, @karol-bisztyga wrote:

I put up a follow-up for avoiding using compile-time flags D3872

To see why it comes as a follow-up, please see https://phabricator.ashoat.com/D3690#107900

I agree, it really makes sense to do it as a separate diff. But at the same time, I don't agree a followup is the right way to do that. The issue is that we introduce new code here and we know that we need to update it. Writing and reviewing this code is quite wasteful, when it is replaced later. The correct way to do that is to make this change before this diff. I know it is painful to alter the stack, but overall it's a lot more efficient approach.

Not going to bock you on this, but please consider introducing the changes before a diff in future stacks.

services/blob/test/MultiPartUploadTest.cpp
24 ↗	(On Diff #11291)	Why we no longer need to specify the region?

This revision is now accepted and ready to land.Apr 29 2022, 5:17 AM

Agree with @palys-swm on diff ordering

Sure, you're right.

services/blob/test/MultiPartUploadTest.cpp
24 ↗	(On Diff #11291)	We do it: config.region = AWS_REGION; In `AwsTools.cpp`

Closed by commit rCOMMe82deb6b61df: [services] Dev mode - Set up Blob service (authored by • karol). · Explain WhyMay 6 2022, 1:46 AM

This revision was automatically updated to reflect the committed changes.

• karol added a commit: rCOMMe82deb6b61df: [services] Dev mode - Set up Blob service.

Revision Contents
Changeset List

Path

Size

services/

blob/

CMakeLists.txt

14 lines

dev/

src/

11 lines

7 lines

DatabaseManager.dev.cpp

MultiPartUploader.dev.cpp

test/

MultiPartUploadTest.cpp

5 lines

Diff 12332

View Options

services/blob/CMakeLists.txt

Show All 17 Lines
import {		import {
deleteOrphanedSessions,		deleteOrphanedSessions,
deleteOldWebSessions,		deleteOldWebSessions,
} from '../deleters/session-deleters.js';		} from '../deleters/session-deleters.js';
import { deleteStaleSIWENonceEntries } from '../deleters/siwe-nonce-deleters.js';		import { deleteStaleSIWENonceEntries } from '../deleters/siwe-nonce-deleters.js';
import { deleteInaccessibleThreads } from '../deleters/thread-deleters.js';		import { deleteInaccessibleThreads } from '../deleters/thread-deleters.js';
import { deleteExpiredUpdates } from '../deleters/update-deleters.js';		import { deleteExpiredUpdates } from '../deleters/update-deleters.js';
import { deleteUnassignedUploads } from '../deleters/upload-deleters.js';		import { deleteUnassignedUploads } from '../deleters/upload-deleters.js';
		import { fetchCallUpdateOlmAccount } from '../updaters/olm-account-updater.js';
		import { validateAccountPrekey } from '../utils/olm-utils.js';

if (cluster.isMaster) {		if (cluster.isMaster) {
schedule.scheduleJob(		schedule.scheduleJob(
'30 3 * * *', // every day at 3:30 AM in the keyserver's timezone		'30 3 * * *', // every day at 3:30 AM in the keyserver's timezone
async () => {		async () => {
try {		try {
// Do everything one at a time to reduce load since we're in no hurry,		// Do everything one at a time to reduce load since we're in no hurry,
// and since some queries depend on previous ones.		// and since some queries depend on previous ones.
▲ Show 20 Lines • Show All 51 Lines • ▼ Show 20 Lines	async () => {
} catch (e) {		} catch (e) {
console.warn(		console.warn(
'encountered error while trying to create daily updates thread',		'encountered error while trying to create daily updates thread',
e,		e,
);		);
}		}
},		},
);		);

		schedule.scheduleJob(
		'0 0 * * *', // every day at midnight in the keyserver's timezone
		async () => {
		try {
		await fetchCallUpdateOlmAccount('content', validateAccountPrekey);
		await fetchCallUpdateOlmAccount('notifications', validateAccountPrekey);
		} catch (e) {
		console.warn('encountered error while trying to validate prekeys', e);
		}
		},
		);
}		}

View Options

services/blob/dev/DatabaseSimulator.h

Show All 17 Lines
import {		import {
deleteOrphanedSessions,		deleteOrphanedSessions,
deleteOldWebSessions,		deleteOldWebSessions,
} from '../deleters/session-deleters.js';		} from '../deleters/session-deleters.js';
import { deleteStaleSIWENonceEntries } from '../deleters/siwe-nonce-deleters.js';		import { deleteStaleSIWENonceEntries } from '../deleters/siwe-nonce-deleters.js';
import { deleteInaccessibleThreads } from '../deleters/thread-deleters.js';		import { deleteInaccessibleThreads } from '../deleters/thread-deleters.js';
import { deleteExpiredUpdates } from '../deleters/update-deleters.js';		import { deleteExpiredUpdates } from '../deleters/update-deleters.js';
import { deleteUnassignedUploads } from '../deleters/upload-deleters.js';		import { deleteUnassignedUploads } from '../deleters/upload-deleters.js';
		import { fetchCallUpdateOlmAccount } from '../updaters/olm-account-updater.js';
		import { validateAccountPrekey } from '../utils/olm-utils.js';

if (cluster.isMaster) {		if (cluster.isMaster) {
schedule.scheduleJob(		schedule.scheduleJob(
'30 3 * * *', // every day at 3:30 AM in the keyserver's timezone		'30 3 * * *', // every day at 3:30 AM in the keyserver's timezone
async () => {		async () => {
try {		try {
// Do everything one at a time to reduce load since we're in no hurry,		// Do everything one at a time to reduce load since we're in no hurry,
// and since some queries depend on previous ones.		// and since some queries depend on previous ones.
▲ Show 20 Lines • Show All 51 Lines • ▼ Show 20 Lines	async () => {
} catch (e) {		} catch (e) {
console.warn(		console.warn(
'encountered error while trying to create daily updates thread',		'encountered error while trying to create daily updates thread',
e,		e,
);		);
}		}
},		},
);		);

		schedule.scheduleJob(
		'0 0 * * *', // every day at midnight in the keyserver's timezone
		async () => {
		try {
		await fetchCallUpdateOlmAccount('content', validateAccountPrekey);
		await fetchCallUpdateOlmAccount('notifications', validateAccountPrekey);
		} catch (e) {
		console.warn('encountered error while trying to validate prekeys', e);
		}
		},
		);
}		}

View Options

services/blob/dev/DevTools.h

Show All 17 Lines
import {		import {
deleteOrphanedSessions,		deleteOrphanedSessions,
deleteOldWebSessions,		deleteOldWebSessions,
} from '../deleters/session-deleters.js';		} from '../deleters/session-deleters.js';
import { deleteStaleSIWENonceEntries } from '../deleters/siwe-nonce-deleters.js';		import { deleteStaleSIWENonceEntries } from '../deleters/siwe-nonce-deleters.js';
import { deleteInaccessibleThreads } from '../deleters/thread-deleters.js';		import { deleteInaccessibleThreads } from '../deleters/thread-deleters.js';
import { deleteExpiredUpdates } from '../deleters/update-deleters.js';		import { deleteExpiredUpdates } from '../deleters/update-deleters.js';
import { deleteUnassignedUploads } from '../deleters/upload-deleters.js';		import { deleteUnassignedUploads } from '../deleters/upload-deleters.js';
		import { fetchCallUpdateOlmAccount } from '../updaters/olm-account-updater.js';
		import { validateAccountPrekey } from '../utils/olm-utils.js';

if (cluster.isMaster) {		if (cluster.isMaster) {
schedule.scheduleJob(		schedule.scheduleJob(
'30 3 * * *', // every day at 3:30 AM in the keyserver's timezone		'30 3 * * *', // every day at 3:30 AM in the keyserver's timezone
async () => {		async () => {
try {		try {
// Do everything one at a time to reduce load since we're in no hurry,		// Do everything one at a time to reduce load since we're in no hurry,
// and since some queries depend on previous ones.		// and since some queries depend on previous ones.
▲ Show 20 Lines • Show All 51 Lines • ▼ Show 20 Lines	async () => {
} catch (e) {		} catch (e) {
console.warn(		console.warn(
'encountered error while trying to create daily updates thread',		'encountered error while trying to create daily updates thread',
e,		e,
);		);
}		}
},		},
);		);

		schedule.scheduleJob(
		'0 0 * * *', // every day at midnight in the keyserver's timezone
		async () => {
		try {
		await fetchCallUpdateOlmAccount('content', validateAccountPrekey);
		await fetchCallUpdateOlmAccount('notifications', validateAccountPrekey);
		} catch (e) {
		console.warn('encountered error while trying to validate prekeys', e);
		}
		},
		);
}		}

View Options

services/blob/dev/DevTools.cpp

Show All 17 Lines
import {		import {
deleteOrphanedSessions,		deleteOrphanedSessions,
deleteOldWebSessions,		deleteOldWebSessions,
} from '../deleters/session-deleters.js';		} from '../deleters/session-deleters.js';
import { deleteStaleSIWENonceEntries } from '../deleters/siwe-nonce-deleters.js';		import { deleteStaleSIWENonceEntries } from '../deleters/siwe-nonce-deleters.js';
import { deleteInaccessibleThreads } from '../deleters/thread-deleters.js';		import { deleteInaccessibleThreads } from '../deleters/thread-deleters.js';
import { deleteExpiredUpdates } from '../deleters/update-deleters.js';		import { deleteExpiredUpdates } from '../deleters/update-deleters.js';
import { deleteUnassignedUploads } from '../deleters/upload-deleters.js';		import { deleteUnassignedUploads } from '../deleters/upload-deleters.js';
		import { fetchCallUpdateOlmAccount } from '../updaters/olm-account-updater.js';
		import { validateAccountPrekey } from '../utils/olm-utils.js';

if (cluster.isMaster) {		if (cluster.isMaster) {
schedule.scheduleJob(		schedule.scheduleJob(
'30 3 * * *', // every day at 3:30 AM in the keyserver's timezone		'30 3 * * *', // every day at 3:30 AM in the keyserver's timezone
async () => {		async () => {
try {		try {
// Do everything one at a time to reduce load since we're in no hurry,		// Do everything one at a time to reduce load since we're in no hurry,
// and since some queries depend on previous ones.		// and since some queries depend on previous ones.
▲ Show 20 Lines • Show All 51 Lines • ▼ Show 20 Lines	async () => {
} catch (e) {		} catch (e) {
console.warn(		console.warn(
'encountered error while trying to create daily updates thread',		'encountered error while trying to create daily updates thread',
e,		e,
);		);
}		}
},		},
);		);

		schedule.scheduleJob(
		'0 0 * * *', // every day at midnight in the keyserver's timezone
		async () => {
		try {
		await fetchCallUpdateOlmAccount('content', validateAccountPrekey);
		await fetchCallUpdateOlmAccount('notifications', validateAccountPrekey);
		} catch (e) {
		console.warn('encountered error while trying to validate prekeys', e);
		}
		},
		);
}		}