Page MenuHomePhabricator

[lib] Properly handle hashing in progress
ClosedPublic

Authored by tomek on Apr 26 2024, 6:52 AM.
Tags
None
Referenced Files
Unknown Object (File)
Wed, Nov 13, 7:57 PM
Unknown Object (File)
Tue, Nov 12, 4:46 PM
Unknown Object (File)
Tue, Nov 12, 4:17 PM
Unknown Object (File)
Tue, Nov 12, 3:32 PM
Unknown Object (File)
Tue, Nov 12, 3:24 PM
Unknown Object (File)
Tue, Nov 12, 3:07 PM
Unknown Object (File)
Tue, Nov 12, 2:29 PM
Unknown Object (File)
Tue, Nov 12, 2:24 PM
Subscribers

Details

Summary

When the hashing is in progress, we tell the server that our store hashes are correct. We were concluding that the hashing is in progress by checking if a hash value is null - which was incorrect because it can be null when an entry is missing. The solution was to introduce a function that tells if we can perform a state sync. This function returns false when thread store hashing is in progress and when we perform user state sync with non-authoritative keyservers.

https://linear.app/comm/issue/ENG-7990/state-sync-sometimes-doesnt-fix-the-state

Test Plan

Check if state sync fixes a state with missing thread info.
Also checked if state sync updates a user name, current user name, and entry text.

Diff Detail

Repository
rCOMM Comm
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

tomek requested review of this revision.Apr 26 2024, 7:08 AM

Handle syncing with non-auth keyservers

lib/selectors/socket-selectors.js
155 ↗(On Diff #39545)

If we're not ready to sync the state when the keyserver asks us to, do we make sure to respond to it once we're ready? Or do we have to wait for the keyserver to initiate again?

lib/shared/state-sync/current-user-state-sync-spec.js
27–28 ↗(On Diff #39545)

For this one and for usersStateSyncSpec, it seems like you're using canSyncState to block user state sync with non-authoritative keyservers. But the non-authoritative keyserver will still attempt to initiate the sync for that data by sending it inside serverRequest.hashesToCheck, right? Is there a task that tracks updating the keyserver code so that non-authoritative keyserver stop trying to sync these fields with clients?

lib/shared/state-sync/entries-state-sync-spec.js
45 ↗(On Diff #39545)

Do we not pre-compute hashes for entry sync?

lib/shared/state-sync/state-sync-spec.js
34 ↗(On Diff #39545)

The way this function is named, it feels like it's commenting on whether this type of state can be synced, rather than whether we're ready to perform a sync.

Maybe you can add a comment above this explaining what this function really does, and how it's used? (In some cases to block non-authoritative keyservers, and in other cases to block sending state until our hashes are ready.)

lib/shared/state-sync/users-state-sync-spec.js
45–46 ↗(On Diff #39545)

Do we not pre-compute hashes for user sync?

lib/selectors/socket-selectors.js
155 ↗(On Diff #39545)

We're waiting for the keyserver to try again. The preparation of integrity store hashes could take up to 10 minutes, so the keyserver can try multiple times during that period and it shouldn't make the experience significantly worse by waiting another up to 3 minutes of inactivity (our state sync frequency). We could consider initiating the state sync from the client side when the hashes are ready, but not sure if it is worth it.

lib/shared/state-sync/current-user-state-sync-spec.js
27–28 ↗(On Diff #39545)

But the non-authoritative keyserver will still attempt to initiate the sync for that data by sending it inside serverRequest.hashesToCheck, right?

Yes, that's correct.

Is there a task that tracks updating the keyserver code so that non-authoritative keyserver stop trying to sync these fields with clients?

We don't have a dedicated task, because we believed that the whole goal of syncing user infos without a keyserver will require this change. But I guess we can consider making this simple change as a part of e.g. https://linear.app/comm/issue/ENG-7741/userstore-syncing-in-multi-keyserver-environment and its subtask https://linear.app/comm/issue/ENG-7798/stop-sending-userinfos-from-non-auth-keyservers.

lib/shared/state-sync/entries-state-sync-spec.js
45 ↗(On Diff #39545)

Yes, we compute all the hashes from scratch on every request. The performance was investigated in https://linear.app/comm/issue/ENG-4883/improve-performance-of-id-conversion and the improvement was implemented in https://linear.app/comm/issue/ENG-4959/integritystore-compute-thread-hashes-greedily#comment-15cf1147. I don't see a comment summarising how long the computation takes for the entry store, but I somehow remember that it was orders of magnitude faster than threads (because entries are really simple and threads have multiple levels of props with IDs to convert).

lib/shared/state-sync/state-sync-spec.js
34 ↗(On Diff #39545)

The way this function is named, it feels like it's commenting on whether this type of state can be synced, rather than whether we're ready to perform a sync.

Yes, for one case it tells if we're ready, and for two other cases if we can sync at all. I'll add a comment explaining this.

lib/shared/state-sync/users-state-sync-spec.js
45–46 ↗(On Diff #39545)

Yes, we're pre-computing only the threads' hashes.

I can accept this since I reviewed it closely already, but feel free to set @inka as blocking if you'd like their review as well

lib/shared/state-sync/state-sync-spec.js
34–41 ↗(On Diff #39641)

Minor edits

This revision is now accepted and ready to land.Apr 29 2024, 12:26 PM
This revision was automatically updated to reflect the committed changes.