Page MenuHomePhabricator
Differential D4030

[services] Backup - Implement generating holder
ClosedPublic
Actions

Authored by • karol on May 13 2022, 2:27 AM.
Tags
None
Referenced Files
Unknown Object (File)
Thu, Apr 24, 5:28 PM
Unknown Object (File)
Fri, Apr 18, 3:37 PM
Unknown Object (File)
Fri, Apr 18, 3:04 PM
Unknown Object (File)
Fri, Apr 18, 10:56 AM
Unknown Object (File)
Fri, Apr 18, 6:41 AM
Unknown Object (File)
Fri, Apr 18, 1:52 AM
Unknown Object (File)
Thu, Apr 17, 9:30 PM
Unknown Object (File)
Thu, Apr 17, 2:42 PM
View All Files
Subscribers
• abosh
• adrian
ashoat
atul
tomek

Details

Reviewers
tomek
Commits
rCOMM3667a7b5bd16: [services] Backup - Implement generating holder
Summary

Depends on D4029

https://linear.app/comm/issue/ENG-1128/implement-generating-holders

Test Plan

Services still build

Diff Detail

Repository
rCOMM Comm
Lint
No Lint Coverage
Unit
No Test Coverage

Event Timeline

• karol created this revision.May 13 2022, 2:27 AM
Herald added subscribers: • abosh, atul, • adrian and 2 others. · View Herald TranscriptMay 13 2022, 2:27 AM
Harbormaster returned this revision to the author for changes because remote builds failed.May 13 2022, 2:27 AM
Harbormaster failed remote builds in B9081: Diff 12627!
• karol edited the summary of this revision. (Show Details)May 13 2022, 2:40 AM
• karol edited the test plan for this revision. (Show Details)
• karol added a reviewer: tomek.
• karol added a parent revision: D4029: [services] Backup - New backup - make created consistent.
• karol added a child revision: D4031: [services] Backup - use generateHolder for new backup.
• karol requested review of this revision.May 13 2022, 2:52 AM

services build passed

tomek requested changes to this revision.May 17 2022, 3:35 AM
tomek added inline comments.
services/backup/src/Tools.cpp
24–34 ↗(On Diff #12627)

If we modify the order of arguments we can have only one function with a default value.

Additionally, we can have the same schema for both cases, so that it's easier to parse it in the future (if we ever need to).

This revision now requires changes to proceed.May 17 2022, 3:35 AM
• karol added inline comments.May 17 2022, 11:43 PM
services/backup/src/Tools.cpp
24–34 ↗(On Diff #12627)

Good point, I missed that. But I think it will be more readable to have resourceID next to backupID, so we could do:

std::string generateHolder(
    const std::string &blobHash,
    const std::string &backupID,
    const std::string &resourceID = ""
    ) {
• karol updated this revision to Diff 12837.May 17 2022, 11:56 PM

fix function args

Harbormaster completed remote builds in B9242: Diff 12837.May 18 2022, 12:01 AM
tomek requested changes to this revision.May 18 2022, 2:44 AM
tomek added inline comments.
services/backup/src/Tools.cpp
28–33 ↗(On Diff #12837)

Additionally, we can have the same schema for both cases, so that it's easier to parse it in the future (if we ever need to).

What do you think about the second part of my comment where I suggested that we should have the same number of sections in id, even when resourceID is empty?

This revision now requires changes to proceed.May 18 2022, 2:44 AM
tomek mentioned this in D4032: [services] Backup - use generateHolder for new log.May 18 2022, 2:46 AM
• karol added inline comments.May 19 2022, 2:36 AM
services/backup/src/Tools.cpp
28–33 ↗(On Diff #12837)

At first I thought that aaa::bbb would look weird with :: but the argument about the parsing makes sense. Sorry I didn't implement this right away. I'm going to change this. Thanks.

• karol updated this revision to Diff 12940.May 19 2022, 4:43 AM

remove ID_SEPARATOR from here

• karol planned changes to this revision.May 19 2022, 4:44 AM
• karol updated this revision to Diff 12943.EditedMay 19 2022, 4:47 AM

fix invalid update, generate the same holder no matter if the resourceID is empty or not

Harbormaster completed remote builds in B9338: Diff 12940.May 19 2022, 5:02 AM
Harbormaster completed remote builds in B9341: Diff 12943.May 19 2022, 5:21 AM
tomek accepted this revision.May 20 2022, 4:24 AM
This revision is now accepted and ready to land.May 20 2022, 4:24 AM
Closed by commit rCOMM3667a7b5bd16: [services] Backup - Implement generating holder (authored by • karol). · Explain WhyMay 25 2022, 3:25 AM
This revision was automatically updated to reflect the committed changes.
• karol added a commit: rCOMM3667a7b5bd16: [services] Backup - Implement generating holder.

Revision Contents
Changeset List

PathSize
services/
backup/
src/
Reactors/
server/
12 lines

Diff 12940

View Options

services/backup/src/Reactors/server/CreateNewBackupReactor.cpp

Show First 20 Lines • Show All 487 Lines • ▼ Show 20 Lines attempt_delete_file(
"Failed to delete unencrypted database."); "Failed to delete unencrypted database.");
attempt_rename_file( attempt_rename_file(
temp_encrypted_db_path, temp_encrypted_db_path,
SQLiteQueryExecutor::sqliteFilePath, SQLiteQueryExecutor::sqliteFilePath,
"Failed to move encrypted database to default location."); "Failed to move encrypted database to default location.");
Logger::log("Encryption completed successfully."); Logger::log("Encryption completed successfully.");
} }
auto &SQLiteQueryExecutor::getStorage() {
static auto storage = make_storage(
SQLiteQueryExecutor::sqliteFilePath,
make_index("messages_idx_thread_time", &Message::thread, &Message::time),
make_index("media_idx_container", &Media::container),
make_table(
"drafts",
make_column("key", &Draft::key, unique(), primary_key()),
make_column("text", &Draft::text)),
make_table(
"messages",
make_column("id", &Message::id, unique(), primary_key()),
make_column("local_id", &Message::local_id),
make_column("thread", &Message::thread),
make_column("user", &Message::user),
make_column("type", &Message::type),
make_column("future_type", &Message::future_type),
make_column("content", &Message::content),
make_column("time", &Message::time)),
make_table(
"olm_persist_account",
make_column("id", &OlmPersistAccount::id, unique(), primary_key()),
make_column("account_data", &OlmPersistAccount::account_data)),
make_table(
"olm_persist_sessions",
make_column(
"target_user_id",
&OlmPersistSession::target_user_id,
unique(),
primary_key()),
make_column("session_data", &OlmPersistSession::session_data)),
make_table(
"media",
make_column("id", &Media::id, unique(), primary_key()),
make_column("container", &Media::container),
make_column("thread", &Media::thread),
make_column("uri", &Media::uri),
make_column("type", &Media::type),
make_column("extras", &Media::extras)),
make_table(
"threads",
make_column("id", &Thread::id, unique(), primary_key()),
make_column("type", &Thread::type),
make_column("name", &Thread::name),
make_column("description", &Thread::description),
make_column("color", &Thread::color),
make_column("creation_time", &Thread::creation_time),
make_column("parent_thread_id", &Thread::parent_thread_id),
make_column("containing_thread_id", &Thread::containing_thread_id),
make_column("community", &Thread::community),
make_column("members", &Thread::members),
make_column("roles", &Thread::roles),
make_column("current_user", &Thread::current_user),
make_column("source_message_id", &Thread::source_message_id),
make_column("replies_count", &Thread::replies_count)),
make_table(
"metadata",
make_column("name", &Metadata::name, unique(), primary_key()),
make_column("data", &Metadata::data)));
storage.on_open = on_database_open;
return storage;
}
typedef bool ShouldBeInTransaction; typedef bool ShouldBeInTransaction;
typedef std::pair<std::function<bool(sqlite3 *)>, ShouldBeInTransaction> typedef std::pair<std::function<bool(sqlite3 *)>, ShouldBeInTransaction>
SQLiteMigration; SQLiteMigration;
std::vector<std::pair<uint, SQLiteMigration>> migrations{ std::vector<std::pair<uint, SQLiteMigration>> migrations{
{{1, {create_drafts_table, true}}, {{1, {create_drafts_table, true}},
{2, {rename_threadID_to_key, true}}, {2, {rename_threadID_to_key, true}},
{4, {create_persist_account_table, true}}, {4, {create_persist_account_table, true}},
{5, {create_persist_sessions_table, true}}, {5, {create_persist_sessions_table, true}},
Show All 28 Linesvoid SQLiteQueryExecutor::migrate() const {
int current_user_version = sqlite3_column_int(user_version_stmt, 0); int current_user_version = sqlite3_column_int(user_version_stmt, 0);
sqlite3_finalize(user_version_stmt); sqlite3_finalize(user_version_stmt);
std::stringstream version_msg; std::stringstream version_msg;
version_msg << "db version: " << current_user_version << std::endl; version_msg << "db version: " << current_user_version << std::endl;
Logger::log(version_msg.str()); Logger::log(version_msg.str());
if (current_user_version == 0) {
SQLiteQueryExecutor::getStorage().sync_schema();
Logger::log("Creating new database, syncing structure with ORM storage.");
auto latest_version = migrations.back().first;
std::stringstream update_version;
update_version << "PRAGMA user_version=" << latest_version << ";";
auto update_version_str = update_version.str();
char *error;
sqlite3_exec(db, update_version_str.c_str(), nullptr, nullptr, &error);
if (error) {
std::ostringstream errorStream;
errorStream << "Error saving database version: " << error;
Logger::log(errorStream.str());
marcinUnsubmitted
Not Done
Inline Actions

How is application going to function if this error occurs? In most places we the runtime error to kill the app if database operation fails since application is not going to work correctly. I just want to know whether it is also the case here.

marcin: How is application going to function if this error occurs? In most places we the runtime error…
sqlite3_free(error);
}
return;
}
for (const auto &[idx, migration] : migrations) { for (const auto &[idx, migration] : migrations) {
if (idx <= current_user_version) { if (idx <= current_user_version) {
continue; continue;
} }
const auto &[applyMigration, shouldBeInTransaction] = migration; const auto &[applyMigration, shouldBeInTransaction] = migration;
std::stringstream migration_msg; std::stringstream migration_msg;
Show All 28 Linesvoid SQLiteQueryExecutor::assign_encryption_key() {
CommSecureStore commSecureStore{}; CommSecureStore commSecureStore{};
std::string encryptionKey = comm::crypto::Tools::generateRandomHexString( std::string encryptionKey = comm::crypto::Tools::generateRandomHexString(
SQLiteQueryExecutor::sqlcipherEncryptionKeySize); SQLiteQueryExecutor::sqlcipherEncryptionKeySize);
commSecureStore.set( commSecureStore.set(
SQLiteQueryExecutor::secureStoreEncryptionKeyID, encryptionKey); SQLiteQueryExecutor::secureStoreEncryptionKeyID, encryptionKey);
SQLiteQueryExecutor::encryptionKey = encryptionKey; SQLiteQueryExecutor::encryptionKey = encryptionKey;
} }
auto &SQLiteQueryExecutor::getStorage() {
static auto storage = make_storage(
SQLiteQueryExecutor::sqliteFilePath,
make_index("messages_idx_thread_time", &Message::thread, &Message::time),
make_index("media_idx_container", &Media::container),
make_table(
"drafts",
make_column("key", &Draft::key, unique(), primary_key()),
make_column("text", &Draft::text)),
make_table(
"messages",
make_column("id", &Message::id, unique(), primary_key()),
make_column("local_id", &Message::local_id),
make_column("thread", &Message::thread),
make_column("user", &Message::user),
make_column("type", &Message::type),
make_column("future_type", &Message::future_type),
make_column("content", &Message::content),
make_column("time", &Message::time)),
make_table(
"olm_persist_account",
make_column("id", &OlmPersistAccount::id, unique(), primary_key()),
make_column("account_data", &OlmPersistAccount::account_data)),
make_table(
"olm_persist_sessions",
make_column(
"target_user_id",
&OlmPersistSession::target_user_id,
unique(),
primary_key()),
make_column("session_data", &OlmPersistSession::session_data)),
make_table(
"media",
make_column("id", &Media::id, unique(), primary_key()),
make_column("container", &Media::container),
make_column("thread", &Media::thread),
make_column("uri", &Media::uri),
make_column("type", &Media::type),
make_column("extras", &Media::extras)),
make_table(
"threads",
make_column("id", &Thread::id, unique(), primary_key()),
make_column("type", &Thread::type),
make_column("name", &Thread::name),
make_column("description", &Thread::description),
make_column("color", &Thread::color),
make_column("creation_time", &Thread::creation_time),
make_column("parent_thread_id", &Thread::parent_thread_id),
make_column("containing_thread_id", &Thread::containing_thread_id),
make_column("community", &Thread::community),
make_column("members", &Thread::members),
make_column("roles", &Thread::roles),
make_column("current_user", &Thread::current_user),
make_column("source_message_id", &Thread::source_message_id),
make_column("replies_count", &Thread::replies_count)),
make_table(
"metadata",
make_column("name", &Metadata::name, unique(), primary_key()),
make_column("data", &Metadata::data)));
storage.on_open = on_database_open;
return storage;
}
void SQLiteQueryExecutor::initialize(std::string &databasePath) { void SQLiteQueryExecutor::initialize(std::string &databasePath) {
std::call_once(SQLiteQueryExecutor::initialized, [&databasePath]() { std::call_once(SQLiteQueryExecutor::initialized, [&databasePath]() {
SQLiteQueryExecutor::sqliteFilePath = databasePath; SQLiteQueryExecutor::sqliteFilePath = databasePath;
CommSecureStore commSecureStore{}; CommSecureStore commSecureStore{};
folly::Optional<std::string> maybeEncryptionKey = folly::Optional<std::string> maybeEncryptionKey =
commSecureStore.get(SQLiteQueryExecutor::secureStoreEncryptionKeyID); commSecureStore.get(SQLiteQueryExecutor::secureStoreEncryptionKeyID);
if (file_exists(databasePath) && maybeEncryptionKey) { if (file_exists(databasePath) && maybeEncryptionKey) {
▲ Show 20 Lines • Show All 277 Lines • Show Last 20 Lines