Page MenuHomePhabricator
Differential D14517

[SQLite] implement method to copy table between databases
ClosedPublic
Actions

Authored by kamil on Thu, Apr 3, 2:37 AM.
Tags
None
Referenced Files
F5576152: D14517.id47639.diff
Sat, Apr 12, 6:36 PM
Unknown Object (File)
Fri, Apr 11, 4:11 AM
Unknown Object (File)
Thu, Apr 10, 11:35 PM
Unknown Object (File)
Thu, Apr 10, 6:34 PM
Unknown Object (File)
Wed, Apr 9, 8:29 PM
Unknown Object (File)
Wed, Apr 9, 12:01 PM
Unknown Object (File)
Wed, Apr 9, 1:39 AM
Unknown Object (File)
Wed, Apr 9, 12:30 AM
View All 20 Files
Subscribers
ashoat
tomek

Details

Reviewers
bartek
Commits
rCOMM277bd9ad9879: [SQLite] implement method to copy table between databases
Summary

ENG-10489.

Depends on D14516

Test Plan

Tested later in the stack, existing tables (e.g. olm_persist_account) are not overwritten during restore

Diff Detail

Repository
rCOMM Comm
Lint
No Lint Coverage
Unit
No Test Coverage

Event Timeline

kamil created this revision.Thu, Apr 3, 2:37 AM
kamil held this revision as a draft.
Herald added subscribers: tomek, ashoat. · View Herald TranscriptThu, Apr 3, 2:37 AM
kamil added a child revision: D14518: [SQLite] unify `restoreFromMainCompaction` on platforms.Thu, Apr 3, 2:42 AM
Harbormaster completed remote builds in B33872: Diff 47622.Thu, Apr 3, 2:55 AM
kamil published this revision for review.Thu, Apr 3, 3:10 AM
kamil added inline comments.
native/cpp/CommonCpp/DatabaseManagers/SQLiteQueryExecutor.cpp
3313 ↗(On Diff #47622)

We need to use this only on not-encrypted databases, see next diff in the stack

3317 ↗(On Diff #47622)

We want OR IGNORE statement to skip duplicated primary keys, the example is a thread with Ashoat, which is created for each user (we won't include thin thread data in backup, but we do now, so without this each copy statement could fail)

native/cpp/CommonCpp/DatabaseManagers/SQLiteQueryExecutor.h
58 ↗(On Diff #47622)

We might want to have a custom copy method (by specifying fields), e.g. for messages where we have one generated column

bartek added a comment.Thu, Apr 3, 3:50 AM

Don't we want this method to take a vector of tables to copy? Looking at D14519, I see you're running this in a loop, so the ATTACH DATABASE ..., DETACH statements are repeated for each table

kamil planned changes to this revision.Thu, Apr 3, 3:55 AM
In D14517#403050, @bartek wrote:

Don't we want this method to take a vector of tables to copy? Looking at D14519, I see you're running this in a loop, so the ATTACH DATABASE ..., DETACH statements are repeated for each table

Good call

kamil updated this revision to Diff 47639.Thu, Apr 3, 5:20 AM

address review

Harbormaster completed remote builds in B33889: Diff 47639.Thu, Apr 3, 5:39 AM
bartek accepted this revision.Thu, Apr 3, 7:25 AM
This revision is now accepted and ready to land.Thu, Apr 3, 7:25 AM
Closed by commit rCOMM277bd9ad9879: [SQLite] implement method to copy table between databases (authored by kamil). · Explain WhyMon, Apr 7, 3:30 AM
This revision was automatically updated to reflect the committed changes.
kamil added a commit: rCOMM277bd9ad9879: [SQLite] implement method to copy table between databases.

Revision Contents
Changeset List

PathSize
native/
cpp/
CommonCpp/
DatabaseManagers/
4 lines
22 lines

Diff 47639

View Options

native/cpp/CommonCpp/DatabaseManagers/SQLiteQueryExecutor.h

Show All 12 Lines
import { import {
createAsyncMigrate, createAsyncMigrate,
type StorageMigrationFunction, type StorageMigrationFunction,
} from 'lib/shared/create-async-migrate.js'; } from 'lib/shared/create-async-migrate.js';
import type { import type {
KeyserverInfo, KeyserverInfo,
KeyserverStore, KeyserverStore,
} from 'lib/types/keyserver-types.js'; } from 'lib/types/keyserver-types.js';
import { cookieTypes } from 'lib/types/session-types.js';
import { import {
defaultConnectionInfo, defaultConnectionInfo,
type ConnectionInfo, type ConnectionInfo,
} from 'lib/types/socket-types.js'; } from 'lib/types/socket-types.js';
import { parseCookies } from 'lib/utils/cookie-utils.js';
import { isDev } from 'lib/utils/dev-utils.js'; import { isDev } from 'lib/utils/dev-utils.js';
import { import {
generateIDSchemaMigrationOpsForDrafts, generateIDSchemaMigrationOpsForDrafts,
convertDraftStoreToNewIDSchema, convertDraftStoreToNewIDSchema,
} from 'lib/utils/migration-utils.js'; } from 'lib/utils/migration-utils.js';
import { ashoatKeyserverID } from 'lib/utils/validation-utils.js'; import { ashoatKeyserverID } from 'lib/utils/validation-utils.js';
import commReduxStorageEngine from './comm-redux-storage-engine.js'; import commReduxStorageEngine from './comm-redux-storage-engine.js';
▲ Show 20 Lines • Show All 92 Lines • ▼ Show 20 Lines [5]: async state => {
await databaseModule.schedule({ await databaseModule.schedule({
type: workerRequestMessageTypes.PROCESS_STORE_OPERATIONS, type: workerRequestMessageTypes.PROCESS_STORE_OPERATIONS,
storeOperations: { draftStoreOperations }, storeOperations: { draftStoreOperations },
}); });
return state; return state;
}, },
[6]: async (state: AppState): Promise<AppState> => {
const params = parseCookies(document.cookie);
let cookie = null;
if (params[cookieTypes.USER]) {
cookie = `${cookieTypes.USER}=${params[cookieTypes.USER]}`;
} else if (params[cookieTypes.ANONYMOUS]) {
cookie = `${cookieTypes.ANONYMOUS}=${params[cookieTypes.ANONYMOUS]}`;
}
return {
...state,
keyserverStore: {
...state.keyserverStore,
keyserverInfos: {
...state.keyserverStore.keyserverInfos,
[ashoatKeyserverID]: {
...state.keyserverStore.keyserverInfos[ashoatKeyserverID],
cookie,
},
},
},
};
},
}; };
const persistWhitelist = [ const persistWhitelist = [
'enabledApps', 'enabledApps',
'deviceID', 'deviceID',
'cryptoStore', 'cryptoStore',
'notifPermissionAlertInfo', 'notifPermissionAlertInfo',
'commServicesAccessToken', 'commServicesAccessToken',
▲ Show 20 Lines • Show All 96 Lines • ▼ Show 20 Linesconst persistConfig: PersistConfig = {
whitelist: isSQLiteSupported() whitelist: isSQLiteSupported()
? persistWhitelist ? persistWhitelist
: [...persistWhitelist, 'draftStore'], : [...persistWhitelist, 'draftStore'],
migrate: (createAsyncMigrate( migrate: (createAsyncMigrate(
migrations, migrations,
{ debug: isDev }, { debug: isDev },
migrateStorageToSQLite, migrateStorageToSQLite,
): any), ): any),
version: 5, version: 6,
transforms: [keyserverStoreTransform], transforms: [keyserverStoreTransform],
}; };
export { persistConfig }; export { persistConfig };
View Options

native/cpp/CommonCpp/DatabaseManagers/SQLiteQueryExecutor.cpp

Show All 12 Lines
import { import {
createAsyncMigrate, createAsyncMigrate,
type StorageMigrationFunction, type StorageMigrationFunction,
} from 'lib/shared/create-async-migrate.js'; } from 'lib/shared/create-async-migrate.js';
import type { import type {
KeyserverInfo, KeyserverInfo,
KeyserverStore, KeyserverStore,
} from 'lib/types/keyserver-types.js'; } from 'lib/types/keyserver-types.js';
import { cookieTypes } from 'lib/types/session-types.js';
import { import {
defaultConnectionInfo, defaultConnectionInfo,
type ConnectionInfo, type ConnectionInfo,
} from 'lib/types/socket-types.js'; } from 'lib/types/socket-types.js';
import { parseCookies } from 'lib/utils/cookie-utils.js';
import { isDev } from 'lib/utils/dev-utils.js'; import { isDev } from 'lib/utils/dev-utils.js';
import { import {
generateIDSchemaMigrationOpsForDrafts, generateIDSchemaMigrationOpsForDrafts,
convertDraftStoreToNewIDSchema, convertDraftStoreToNewIDSchema,
} from 'lib/utils/migration-utils.js'; } from 'lib/utils/migration-utils.js';
import { ashoatKeyserverID } from 'lib/utils/validation-utils.js'; import { ashoatKeyserverID } from 'lib/utils/validation-utils.js';
import commReduxStorageEngine from './comm-redux-storage-engine.js'; import commReduxStorageEngine from './comm-redux-storage-engine.js';
▲ Show 20 Lines • Show All 92 Lines • ▼ Show 20 Lines [5]: async state => {
await databaseModule.schedule({ await databaseModule.schedule({
type: workerRequestMessageTypes.PROCESS_STORE_OPERATIONS, type: workerRequestMessageTypes.PROCESS_STORE_OPERATIONS,
storeOperations: { draftStoreOperations }, storeOperations: { draftStoreOperations },
}); });
return state; return state;
}, },
[6]: async (state: AppState): Promise<AppState> => {
const params = parseCookies(document.cookie);
let cookie = null;
if (params[cookieTypes.USER]) {
cookie = `${cookieTypes.USER}=${params[cookieTypes.USER]}`;
} else if (params[cookieTypes.ANONYMOUS]) {
cookie = `${cookieTypes.ANONYMOUS}=${params[cookieTypes.ANONYMOUS]}`;
}
return {
...state,
keyserverStore: {
...state.keyserverStore,
keyserverInfos: {
...state.keyserverStore.keyserverInfos,
[ashoatKeyserverID]: {
...state.keyserverStore.keyserverInfos[ashoatKeyserverID],
cookie,
},
},
},
};
},
}; };
const persistWhitelist = [ const persistWhitelist = [
'enabledApps', 'enabledApps',
'deviceID', 'deviceID',
'cryptoStore', 'cryptoStore',
'notifPermissionAlertInfo', 'notifPermissionAlertInfo',
'commServicesAccessToken', 'commServicesAccessToken',
▲ Show 20 Lines • Show All 96 Lines • ▼ Show 20 Linesconst persistConfig: PersistConfig = {
whitelist: isSQLiteSupported() whitelist: isSQLiteSupported()
? persistWhitelist ? persistWhitelist
: [...persistWhitelist, 'draftStore'], : [...persistWhitelist, 'draftStore'],
migrate: (createAsyncMigrate( migrate: (createAsyncMigrate(
migrations, migrations,
{ debug: isDev }, { debug: isDev },
migrateStorageToSQLite, migrateStorageToSQLite,
): any), ): any),
version: 5, version: 6,
transforms: [keyserverStoreTransform], transforms: [keyserverStoreTransform],
}; };
export { persistConfig }; export { persistConfig };