Page MenuHomePhabricator
Differential D9343

[keyserver] Test impact of ThreadStore reductions on Atul's ThreadStore
ClosedPublic
Actions

Authored by ashoat on Sep 29 2023, 1:34 PM.
Tags
None
Referenced Files
Unknown Object (File)
Fri, Oct 4, 4:06 PM
Unknown Object (File)
Wed, Sep 18, 2:00 AM
Unknown Object (File)
Sun, Sep 8, 9:52 AM
Unknown Object (File)
Sun, Sep 8, 9:52 AM
Unknown Object (File)
Sun, Sep 8, 9:52 AM
Unknown Object (File)
Sun, Sep 8, 9:49 AM
Unknown Object (File)
Sat, Sep 7, 8:02 PM
Unknown Object (File)
Aug 28 2024, 4:07 AM
View All 53 Files
Subscribers
None

Details

Reviewers
atul
tomek
Commits
rCOMMd2fdf5398d9e: [keyserver] Test impact of ThreadStore reductions on Atul's ThreadStore
Summary

This diff stack introduces a script that tests various different approaches to reducing the size of the ThreadStore. We may or may not land it... the goal is just to get some numbers.

My ThreadStore has some unique quirks, so I figured it would be good to test at least one other.

Depends on D9342

Test Plan

Final results:

before reductions, Ashoat's ThreadStore is 74509866 bytes large (802748 bytes compressed)
after [A] filtering out old sidebars, Ashoat's ThreadStore is 28795130 bytes large (469736 bytes compressed)
after [B] improving encoding, Ashoat's ThreadStore is 11379070 bytes large (378324 bytes compressed)
after [C] introducing isAdminRole, Ashoat's ThreadStore is 14839491 bytes large (469148 bytes compressed)
after [D] introducing ThreadRolePermissionsStore, Ashoat's ThreadStore is 5640441 bytes large (322220 bytes compressed)
after [A] and [B], Ashoat's ThreadStore is 5711706 bytes large (232280 bytes compressed)
after [A] and [C], Ashoat's ThreadStore is 10047103 bytes large (287884 bytes compressed)
after [A] and [D], Ashoat's ThreadStore is 3064171 bytes large (199800 bytes compressed)
after [B] and [C], Ashoat's ThreadStore is 6993311 bytes large (346120 bytes compressed)
after [B] and [D], Ashoat's ThreadStore is 5632940 bytes large (323304 bytes compressed)
after [C] and [D], Ashoat's ThreadStore is 5640402 bytes large (322508 bytes compressed)
after [A], [B], and [C], Ashoat's ThreadStore is 4121983 bytes large (218956 bytes compressed)
after [A], [B], and [D], Ashoat's ThreadStore is 3058364 bytes large (200104 bytes compressed)
after [A], [C], and [D], Ashoat's ThreadStore is 3064132 bytes large (200072 bytes compressed)
after [B], [C], and [D], Ashoat's ThreadStore is 5632901 bytes large (323400 bytes compressed)
after [A], [B], [C], and [D], Ashoat's ThreadStore is 3058325 bytes large (200056 bytes compressed)
before reductions, Atul's ThreadStore is 45113868 bytes large (285084 bytes compressed)
after [A] filtering out old sidebars, Atul's ThreadStore is 4667972 bytes large (52036 bytes compressed)
after [B] improving encoding, Atul's ThreadStore is 5567741 bytes large (135580 bytes compressed)
after [C] introducing isAdminRole, Atul's ThreadStore is 4429463 bytes large (165248 bytes compressed)
after [D] introducing ThreadRolePermissionsStore, Atul's ThreadStore is 2460260 bytes large (115260 bytes compressed)
after [A] and [B], Atul's ThreadStore is 633244 bytes large (27396 bytes compressed)
after [A] and [C], Atul's ThreadStore is 684532 bytes large (37800 bytes compressed)
after [A] and [D], Atul's ThreadStore is 297540 bytes large (24240 bytes compressed)
after [B] and [C], Atul's ThreadStore is 2694710 bytes large (117604 bytes compressed)
after [B] and [D], Atul's ThreadStore is 2455381 bytes large (115264 bytes compressed)
after [C] and [D], Atul's ThreadStore is 2460251 bytes large (115628 bytes compressed)
after [A], [B], and [C], Atul's ThreadStore is 345579 bytes large (24436 bytes compressed)
after [A], [B], and [D], Atul's ThreadStore is 293864 bytes large (23964 bytes compressed)
after [A], [C], and [D], Atul's ThreadStore is 297531 bytes large (24172 bytes compressed)
after [B], [C], and [D], Atul's ThreadStore is 2455372 bytes large (115352 bytes compressed)
after [A], [B], [C], and [D], Atul's ThreadStore is 293855 bytes large (23996 bytes compressed)

NOTE: I've been tested this on a copy of my keyserver's production database from a couple days back. To make sure the data on approach A is accurate, I replaced Date.now() with the timestamp of the last message in the database. As a result, the numbers above might be slightly different from the numbers in earlier diffs.

Diff Detail

Repository
rCOMM Comm
Lint
No Lint Coverage
Unit
No Test Coverage

Event Timeline

ashoat created this revision.Sep 29 2023, 1:34 PM
ashoat added a comment.Sep 29 2023, 1:54 PM

Here's the data in a nice table format.

Description of approaches
ApproachDescription
[A]filter out sidebars older than a week
[B]improve encoding of permissions blobs by (1) removing false entries entirely, (2) using 1 instead of true, (3) no longer including the source parameter, (4) using minimally small strings as keys
[C]introduce isAdminRole, which allows us to remove permissions blobs from members entirely. also omit isDefaultRole when it's false
[D]remove permissions blobs from members and currentUser on the assumption they can be derived from the role blobs. dedup all role permissions blobs into a separate store (only 11 unique in Ashoat's store)
Ashoat's ThreadStore
AproachUncompressed resultBrotli-compressed result
before reductions71.06 MiB784 KiB
[A]27.46 MiB459 KiB
[B]10.85 MiB369 KiB
[C]14.15 MiB458 KiB
[D]5.38 MiB315 KiB
[A], [B]5.45 MiB227 KiB
[A], [C]9.58 MiB281 KiB
[A], [D]2.92 MiB195 KiB
[B], [C]6.67 MiB338 KiB
[B], [D]5.37 MiB316 KiB
[C], [D]5.38 MiB315 KiB
[A], [B], [C]3.93 MiB214 KiB
[A], [B], [D]2.92 MiB195 KiB
[A], [C], [D]2.92 MiB195 KiB
[B], [C], [D]5.37 MiB316 KiB
[A], [B], [C], [D]2.92 MiB195 KiB
Atul's ThreadStore
AproachUncompressed resultBrotli-compressed result
before reductions43.02 MiB278 KiB
[A]4.45 MiB51 KiB
[B]5.31 MiB132 KiB
[C]4.22 MiB161 KiB
[D]2.35 MiB113 KiB
[A], [B]0.60 MiB27 KiB
[A], [C]0.65 MiB37 KiB
[A], [D]0.28 MiB24 KiB
[B], [C]2.57 MiB115 KiB
[B], [D]2.34 MiB113 KiB
[C], [D]2.34 MiB113 KiB
[A], [B], [C]0.33 MiB24 KiB
[A], [B], [D]0.28 MiB23 KiB
[A], [C], [D]0.28 MiB24 KiB
[B], [C], [D]2.34 MiB113 KiB
[A], [B], [C], [D]0.28 MiB23 KiB
Harbormaster completed remote builds in B22937: Diff 31576.Sep 29 2023, 1:59 PM
ashoat requested review of this revision.Sep 29 2023, 1:59 PM
atul accepted this revision.Sep 29 2023, 7:39 PM

Whether or not this stack gets landed, was super helpful to understand how each solution alone/in combination with others would affect size of ThreadStore

This revision is now accepted and ready to land.Sep 29 2023, 7:39 PM
Closed by commit rCOMMd2fdf5398d9e: [keyserver] Test impact of ThreadStore reductions on Atul's ThreadStore (authored by ashoat). · Explain WhySep 30 2023, 5:28 PM
This revision was automatically updated to reflect the committed changes.
ashoat added a commit: rCOMMd2fdf5398d9e: [keyserver] Test impact of ThreadStore reductions on Atul's ThreadStore.

Revision Contents
Changeset List

PathSize
keyserver/
src/
scripts/
41 lines

Diff 31576

View Options

keyserver/src/scripts/test-threadstore-reductions.js

Loading...