iOS emulator takes about 1ms per one thread and the speed should be O(n). In the case from the linear task where the state check took 53s, there was about 2000 threads, so logging in would take additional 2s

This is concerning. As an approximation of the size of my ThreadStore, I ran this query on my production keyserver, and got 6977 results. Based on 1ms per thread, it looks like this would take 7 seconds for me.

I don't think we can move forward with a solution that does all of this processing on login. I also agree that doing it during the state check doesn't make sense either. Instead, for a fresh log-in I think we should process it in 100ms batches (maybe one batch every 5s) until everything is ready. If the keyserver asks for a state check before then, we should have some mechanism to wait until all of the batches are complete.

I also tested another idea: let's do all "batch hashing" (where we hash all items of a store) on the keyserver side. So during login the keyserver would send client all thread hashes. The client would only update the hashes during thread ops. That means we should optimise the keyserver-side calculations, so we should do the schema conversion on the client. I tested it and got:

• Do the hashing (without conversion) on the keyserver during login => I got ~0.5s for 7300 threads (not sure how that would translate to the actual keyserver though)
• From previous testing: updating one hash with conversion on native should take about ~10ms, but it should be a few threads at most at one time

@ashoat any thoughts?

Separately if we decide to to go with "staggered hashing" I have question regarding:

If the keyserver asks for a state check before then, we should have some mechanism to wait until all of the batches are complete.

Do you mean just blocking until all hashes are complete (like it is currently) or something smarter (e.g. we send a request to the server to delay the state check)?

Do you mean just blocking until all hashes are complete (like it is currently) or something smarter (e.g. we send a request to the server to delay the state check)?

I wasn't thinking of anything smarter. Just want to make sure that the state check on the keyserver side doesn't time out due to delays on the client.

In D9265#272618, @michal wrote:

I also tested another idea: let's do all "batch hashing" (where we hash all items of a store) on the keyserver side. So during login the keyserver would send client all thread hashes. The client would only update the hashes during thread ops. That means we should optimise the keyserver-side calculations, so we should do the schema conversion on the client. I tested it and got:

• Do the hashing (without conversion) on the keyserver during login => I got ~0.5s for 7300 threads (not sure how that would translate to the actual keyserver though)
• From previous testing: updating one hash with conversion on native should take about ~10ms, but it should be a few threads at most at one time

@ashoat any thoughts?

The downside there would be increasing the size of that initial payload... it's already quite expensive, which makes login on native and loading the website take very long.

I'm open to it but I figure doing it on the client doesn't need to be an issue if we make sure to space it out properly. Also curious for @atul's thoughts given he's spent so much time on performance recently.

Is this new IntegrityStore persisted via redux-persist? If so, it's very unfortunate that we're adding more O(n) data to redux-persist... this decision has set back our efforts to move all O(n) data to SQLite.

When we know we're making an effort to achieve a long-term goal, we should always try to move that goal forward incrementally in all of our work.

Moving the goal *backwards* for expediency is a recipe for never achieving that goal.

At the very least, the following needs to be done:

1. Update @kamil's Comm Storage document to reflect the addition of new O(n) data to redux-persist
2. Create a follow-up task to address this

Apologies if I'm misintepreting here, or if I've previously indicated that using redux-persist here is acceptable.

1. I've updated the notion doc
2. Created tasks here: ENG-5235 (web), ENG-5234 (native) as children of "Migrate data to SQLite"

Separately: while integrityStore shouldn't be a big problem for performance as it stores just a string + number for each thread, maybe it would be a good idea to add a code comment with this idea ("store O(n) data in sqlite")? Currently it's only stated in notion doc and some linear tasks which aren't really discoverable.