If I understand correctly this will try to remove S3 blobs that might not exist (for holders can exist without having a blob uploaded and the logic will run for them too). Is this expected?

LGTM, potentially we could add these optimisations:

1. Return impl Iterator<> instead of Vec
2. Return &'self str in blobs_to_find_holders

but I haven't analysed it very thoroughly so there might be lifetime/ergonomics issues

Rebase

Rebase

Rebase

Create a new function in spec that combines fetch and getServerInfosHash so flow doesn't complain

Fixed an issue where the integrity handler would run before the threads were fetched from db.

Change dependencies arrays, fix interval, clearTimeout as effect cleanup, fix array type.

Rebase

Comments

Update to new threadHashingStatus

Added comments and migrations. Extracted array chunking logic to utils. Fix issues with new action and types.

We've talked about this, but heads-up that in terms of sequencing, I'm thinking of that one as a sort of a "follow-up" to the multi-keyserver launch... we could launch with the expectation that users input keyserver URLs manually.

Setting as "plan changes" so it's not landed by mistake

Rebase

This should probably be error! instead of debug! right?

Spacing out hashes calculations on web page load and login/register.

I just realized a problem with approach: we don't want to send all of the input to all of the keyservers. For example in D9240, in fetchSingleMostRecentMessagesFromThreads we want to send each keyserver only the ids that it "owns". Otherwise we could expose private date to other keyservers. We need some logic to split a single request into multiple request, one for each keyserver.

Requesting changes because I have a few smaller comments

Yeah, the electron clients are bit bigger problem... There is a task for the reload prompt: ENG-961 but it wasn't prioritized. For this task I guess we could just leave this diff on phab for some time to let clients migrate? For the desktop app we could even make a "dummy" release to force a reload but not sure if that's worth it.

Hey, I run yarn cleaninstall and got a warning:

Warning: patch-package detected a patch file version mismatch

Is my understanding correct that there is still a toplevel connection field in redux state? Should we remove it inside of this migration?

I also tested another idea: let's do all "batch hashing" (where we hash all items of a store) on the keyserver side. So during login the keyserver would send client all thread hashes. The client would only update the hashes during thread ops. That means we should optimise the keyserver-side calculations, so we should do the schema conversion on the client. I tested it and got:

• Do the hashing (without conversion) on the keyserver during login => I got ~0.5s for 7300 threads (not sure how that would translate to the actual keyserver though)
• From previous testing: updating one hash with conversion on native should take about ~10ms, but it should be a few threads at most at one time

Alternative approach in D9265

Alternative approach in D9265

Yeah, you are correct. Basically if the value you would add previously put in website-responders is:

• some default value than it should go to web/redux/default-state
• value calculated on the keyserver than it should go in getInitialReduxStateResponder (+ some dummy default in default-state so flow doesn't complain)

Selector performance:
From looking at the code, we always select by state.threadStore.threadInfos, so there should be no issue. I've put up the next diff (D9256) which contains the actual changes to the threadStore: the newly added threadHashes are always updated at the same time as threadInfos inside of the threadStoreOpsHandlers.

Ok, seems like constant_time_eq checks the lengths inside, from looking at the code

Just some nits

I'm not really familiar with this stuff, so I might be completely wrong but:

Nice!

This does not handle pending threads (more precisely sidebars created from a message). It's might be fine depending on the use case but I it feel like it should have a code comment in that case. Also agree with @tomek comments.

Overall looks good but I have a question. In the test plan you write:

const searchMessagesActionObj: KeyserverCall<
  [SearchMessagesRequest],
  SearchMessagesResponse,
> = {
  actionFunc: searchMessages,
  config: { keyserverSelection: 'fanout' },
};

but shouldn't the return type be [SearchMessagesResponse] as it's a fanout action?

Made a task about validators ENG-4982 because I have more context on them.

Unfortunate that we can't use ReturnType<T> and Parameters<T> from new versions of flow, might simplify it a little

Return instead of modifying

Fixes

Fixes

Small fixes and early return

Set DatabaseModule fields to optional and add invariants

Improved error handling in D9141. It will bubble up any errors to the error boundary.

Changed the component to function instead of arrow function. Added the error handling so any errors from the promise will bubble up to ErrorBoundary and display the "Something went wrong, please reload" page. We don't really have a any other good way to handle errors here.

Fixed window.status -> this.status issue