Page MenuHomePhabricator

Implement restore from backup logs
ClosedPublic

Authored by marcin on Jan 31 2024, 2:44 PM.
Tags
None
Referenced Files
F3376330: D10905.diff
Tue, Nov 26, 11:24 PM
Unknown Object (File)
Sat, Nov 23, 9:42 AM
Unknown Object (File)
Fri, Nov 8, 5:38 AM
Unknown Object (File)
Fri, Nov 8, 4:35 AM
Unknown Object (File)
Fri, Nov 8, 2:45 AM
Unknown Object (File)
Fri, Nov 8, 2:44 AM
Unknown Object (File)
Fri, Nov 8, 2:42 AM
Unknown Object (File)
Wed, Nov 6, 4:26 PM
Subscribers

Details

Summary

This differential implements restore from backup log function on web and native. Backup logs aren't decrypted - I agreed with @michal that log decryption will take place in rust just after downloading log.

Test Plan
  1. Temporarily disable backup log encryption. Replace encryptedLogBytes with logBytes in NativeConnectionManager.cpp in persistLog.
  2. Build native app.
  3. Use the app, create some drafts.
  4. Download backup logs from XCode.
  5. Apply this patch to web: https://gist.github.com/marcinwasowicz/1fd0e17c4808d72467ecf11dca9e0ca3
  6. Find the first log that creates draft and use button introduced by the patch above to trigger restoration.
  7. Refresh web app. Ensure draft created on native is there.

To test native:

  1. Comment out removing backup directory in DatabaseManager::clearSensitiveData().
  2. Comment out code that encrypts logs.
  3. Add a call that creates main compaction restore_backup in backup.rs.
  4. Build the app.
  5. Create compaction by pressing button in backup menu.
  6. Create some drafts.
  7. Log out.
  8. Download app Documents container from XCode and see how many logs are there.
  9. Change restore_backup to restore from all logs files that affect drafts.
  10. Build the app. Log in.
  11. Press button in backup menu to restore from logs
  12. See that drafts are recreated.

In order to test conflict handler repeat steps above but do not restore from all logs in step 9. Basically skip log that inserts draft and apply only those that modify it. Ensure no drafts are created but logs in the XCode console appear informing that conflict occurred.

Diff Detail

Repository
rCOMM Comm
Branch
marcin/eng-5264
Lint
No Lint Coverage
Unit
No Test Coverage

Event Timeline

Add native implementation I accidentlly forgot

marcin edited the test plan for this revision. (Show Details)
native/cpp/CommonCpp/DatabaseManagers/NativeConnectionManager.cpp
238–240 ↗(On Diff #36487)

This temporarily disables logs capture for the time of restoration.

native/cpp/CommonCpp/DatabaseManagers/SQLiteConnectionManager.cpp
67 ↗(On Diff #36487)

This function is invoked when sqlite3changeset_apply encounters issue when applying change. I highly recommend reviewers to read detailed explanation here, but I will shortly explain what is going on here and why I decided to implement conflict handler like this.

The second argument to conflict handler is the conflict reason and return value from handler dictates how conflicting change is treated. In our case we can have the following conflict reasons:

  • SQLITE_CHANGESET_NOTFOUND - when deleting a row that doesn't exist or when updating the row that doesn't exists. Both of these would happen if we miss a log that inserts relevant row.
  • SQLITE_CHANGESET_CONSTRAINT - when any operation would violate database constraints. This could happen if we miss a log that deletes a row in a table with unique key and then try to insert another row with the same value in this field.
  • SQLITE_CHANGESET_CONFLICT - this happens if we try to insert data but row with the same primary key already exists. This can happen if we miss deletion log. Fortunately we have an easy way to recover from this one. By returning SQLITE_CHANGESET_REPLACE we will make sqlite3shangeset_apply delete previous row and reattempt insertion.

In summary all conflicts we might encounter are due to the fact that we missed some logs. Apart from the last one we don't have an easy automatic way to recover from them. Theoretically we could recover from SQLITE_CHANGESET_NOTFOUND during update by inspecting update statement and constructing insert statement from it. However it would complicate the code significantly. That said I implemented this stack to ensure that no logs are missed in the first place, so in conflict handler I just inspect conflict content, log it and either solve if is the easy case with insert or return SQLITE_CHANGESET_OMIT which makes sqlite3changeset_apply ignore the change.

This revision is now accepted and ready to land.Feb 2 2024, 10:23 AM

Can you add a test case when you test backupLogRestoreConflictHandler?

Rebase + rust bindings for restoration from logs

Can you add a test case when you test backupLogRestoreConflictHandler?

Test plan updated.

This revision was automatically updated to reflect the committed changes.