Paths

Table of Contentst

[keyserver] Test impact of ThreadStore reductions on Atul's ThreadStore
ClosedPublic
Actions

Authored by ashoat on Sep 29 2023, 1:34 PM.

Details

Reviewers

atul
tomek

Commits

rCOMMd2fdf5398d9e: [keyserver] Test impact of ThreadStore reductions on Atul's ThreadStore

Summary

This diff stack introduces a script that tests various different approaches to reducing the size of the ThreadStore. We may or may not land it... the goal is just to get some numbers.

My ThreadStore has some unique quirks, so I figured it would be good to test at least one other.

Depends on D9342

Test Plan

Final results:

before reductions, Ashoat's ThreadStore is 74509866 bytes large (802748 bytes compressed)
after [A] filtering out old sidebars, Ashoat's ThreadStore is 28795130 bytes large (469736 bytes compressed)
after [B] improving encoding, Ashoat's ThreadStore is 11379070 bytes large (378324 bytes compressed)
after [C] introducing isAdminRole, Ashoat's ThreadStore is 14839491 bytes large (469148 bytes compressed)
after [D] introducing ThreadRolePermissionsStore, Ashoat's ThreadStore is 5640441 bytes large (322220 bytes compressed)
after [A] and [B], Ashoat's ThreadStore is 5711706 bytes large (232280 bytes compressed)
after [A] and [C], Ashoat's ThreadStore is 10047103 bytes large (287884 bytes compressed)
after [A] and [D], Ashoat's ThreadStore is 3064171 bytes large (199800 bytes compressed)
after [B] and [C], Ashoat's ThreadStore is 6993311 bytes large (346120 bytes compressed)
after [B] and [D], Ashoat's ThreadStore is 5632940 bytes large (323304 bytes compressed)
after [C] and [D], Ashoat's ThreadStore is 5640402 bytes large (322508 bytes compressed)
after [A], [B], and [C], Ashoat's ThreadStore is 4121983 bytes large (218956 bytes compressed)
after [A], [B], and [D], Ashoat's ThreadStore is 3058364 bytes large (200104 bytes compressed)
after [A], [C], and [D], Ashoat's ThreadStore is 3064132 bytes large (200072 bytes compressed)
after [B], [C], and [D], Ashoat's ThreadStore is 5632901 bytes large (323400 bytes compressed)
after [A], [B], [C], and [D], Ashoat's ThreadStore is 3058325 bytes large (200056 bytes compressed)
before reductions, Atul's ThreadStore is 45113868 bytes large (285084 bytes compressed)
after [A] filtering out old sidebars, Atul's ThreadStore is 4667972 bytes large (52036 bytes compressed)
after [B] improving encoding, Atul's ThreadStore is 5567741 bytes large (135580 bytes compressed)
after [C] introducing isAdminRole, Atul's ThreadStore is 4429463 bytes large (165248 bytes compressed)
after [D] introducing ThreadRolePermissionsStore, Atul's ThreadStore is 2460260 bytes large (115260 bytes compressed)
after [A] and [B], Atul's ThreadStore is 633244 bytes large (27396 bytes compressed)
after [A] and [C], Atul's ThreadStore is 684532 bytes large (37800 bytes compressed)
after [A] and [D], Atul's ThreadStore is 297540 bytes large (24240 bytes compressed)
after [B] and [C], Atul's ThreadStore is 2694710 bytes large (117604 bytes compressed)
after [B] and [D], Atul's ThreadStore is 2455381 bytes large (115264 bytes compressed)
after [C] and [D], Atul's ThreadStore is 2460251 bytes large (115628 bytes compressed)
after [A], [B], and [C], Atul's ThreadStore is 345579 bytes large (24436 bytes compressed)
after [A], [B], and [D], Atul's ThreadStore is 293864 bytes large (23964 bytes compressed)
after [A], [C], and [D], Atul's ThreadStore is 297531 bytes large (24172 bytes compressed)
after [B], [C], and [D], Atul's ThreadStore is 2455372 bytes large (115352 bytes compressed)
after [A], [B], [C], and [D], Atul's ThreadStore is 293855 bytes large (23996 bytes compressed)

NOTE: I've been tested this on a copy of my keyserver's production database from a couple days back. To make sure the data on approach A is accurate, I replaced Date.now() with the timestamp of the last message in the database. As a result, the numbers above might be slightly different from the numbers in earlier diffs.

Diff Detail

Repository

rCOMM Comm

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

ashoat created this revision.Sep 29 2023, 1:34 PM

Here's the data in a nice table format.

Description of approaches

Approach	Description
[A]	filter out sidebars older than a week
[B]	improve encoding of permissions blobs by (1) removing false entries entirely, (2) using `1` instead of `true`, (3) no longer including the `source` parameter, (4) using minimally small strings as keys
[C]	introduce `isAdminRole`, which allows us to remove `permissions` blobs from `members` entirely. also omit `isDefaultRole` when it's `false`
[D]	remove `permissions` blobs from `members` and `currentUser` on the assumption they can be derived from the role blobs. dedup all role permissions blobs into a separate store (only 11 unique in Ashoat's store)

Ashoat's ThreadStore

Aproach	Uncompressed result	Brotli-compressed result
before reductions	71.06 MiB	784 KiB
[A]	27.46 MiB	459 KiB
[B]	10.85 MiB	369 KiB
[C]	14.15 MiB	458 KiB
[D]	5.38 MiB	315 KiB
[A], [B]	5.45 MiB	227 KiB
[A], [C]	9.58 MiB	281 KiB
[A], [D]	2.92 MiB	195 KiB
[B], [C]	6.67 MiB	338 KiB
[B], [D]	5.37 MiB	316 KiB
[C], [D]	5.38 MiB	315 KiB
[A], [B], [C]	3.93 MiB	214 KiB
[A], [B], [D]	2.92 MiB	195 KiB
[A], [C], [D]	2.92 MiB	195 KiB
[B], [C], [D]	5.37 MiB	316 KiB
[A], [B], [C], [D]	2.92 MiB	195 KiB

Atul's ThreadStore

Aproach	Uncompressed result	Brotli-compressed result
before reductions	43.02 MiB	278 KiB
[A]	4.45 MiB	51 KiB
[B]	5.31 MiB	132 KiB
[C]	4.22 MiB	161 KiB
[D]	2.35 MiB	113 KiB
[A], [B]	0.60 MiB	27 KiB
[A], [C]	0.65 MiB	37 KiB
[A], [D]	0.28 MiB	24 KiB
[B], [C]	2.57 MiB	115 KiB
[B], [D]	2.34 MiB	113 KiB
[C], [D]	2.34 MiB	113 KiB
[A], [B], [C]	0.33 MiB	24 KiB
[A], [B], [D]	0.28 MiB	23 KiB
[A], [C], [D]	0.28 MiB	24 KiB
[B], [C], [D]	2.34 MiB	113 KiB
[A], [B], [C], [D]	0.28 MiB	23 KiB

Harbormaster completed remote builds in B22937: Diff 31576.Sep 29 2023, 1:59 PM

ashoat requested review of this revision.Sep 29 2023, 1:59 PM

Whether or not this stack gets landed, was super helpful to understand how each solution alone/in combination with others would affect size of ThreadStore

This revision is now accepted and ready to land.Sep 29 2023, 7:39 PM

Closed by commit rCOMMd2fdf5398d9e: [keyserver] Test impact of ThreadStore reductions on Atul's ThreadStore (authored by ashoat). · Explain WhySep 30 2023, 5:28 PM

This revision was automatically updated to reflect the committed changes.

ashoat added a commit: rCOMMd2fdf5398d9e: [keyserver] Test impact of ThreadStore reductions on Atul's ThreadStore.

Revision Contents
Changeset List

Path

Size

keyserver/

src/

scripts/

test-threadstore-reductions.js

41 lines

Diff 31587

View Options

keyserver/src/scripts/test-threadstore-reductions.js

[keyserver] Test impact of ThreadStore reductions on Atul's ThreadStoreClosedPublicActions