Page MenuHomePhabricator
Differential D14485

[web][keyserver] log Olm memory usage changes
ClosedPublic
Actions

Authored by kamil on Thu, Mar 20, 5:37 AM.
Tags
None
Referenced Files
F5086172: D14485.id47498.diff
Sat, Mar 29, 6:49 PM
F5081362: D14485.id47504.diff
Sat, Mar 29, 5:35 AM
Unknown Object (File)
Fri, Mar 28, 7:59 PM
Unknown Object (File)
Thu, Mar 27, 10:10 PM
Unknown Object (File)
Thu, Mar 27, 3:29 PM
Unknown Object (File)
Thu, Mar 27, 2:51 PM
Unknown Object (File)
Wed, Mar 26, 7:56 PM
Unknown Object (File)
Wed, Mar 26, 7:53 PM
View All 20 Files
Subscribers
None

Details

Reviewers
bartek
tomek
ashoat
Commits
rCOMM993689cd4a3d: [web][keyserver] log Olm memory usage changes
Summary

Some context in ENG-10393.

Those metrics are helpful for two reasons:

  1. get_total_memory - gives us insight into total memory allocated, spikes in this value mean that we might have a lot of promises running at once which allocated a lot of resources - I described this theory in ENG-10393#comment-b874b5ec.
  2. get_used_memory - if this value grows over time we're experiencing memory leaks (except for session creation on the web, see below).

For the web, we might expect used memory to grow when using the app because we're keeping sessions in memory, which means after starting a conversation with a new user used memory should grow, but only for the size of the Session object. The logs could help us discover whether the in-memory store isn't too big, and if so - refactor this to avoid keeping all sessions live at once.

Looking at the transform-remove-console config we exclude error so should be fine.

Depends on D14484

Test Plan
  1. Tested cronjob to print some results when needed (after commenting free).
  2. Tested web to print results.

Diff Detail

Repository
rCOMM Comm
Branch
logging
Lint
No Lint Coverage
Unit
No Test Coverage

Event Timeline

kamil created this revision.Thu, Mar 20, 5:37 AM
kamil held this revision as a draft.
kamil edited the summary of this revision. (Show Details)Thu, Mar 20, 5:43 AM
kamil published this revision for review.Thu, Mar 20, 5:45 AM
ashoat accepted this revision.Thu, Mar 20, 5:52 AM
ashoat added inline comments.
lib/utils/olm-memory-utils.js
19
28
This revision is now accepted and ready to land.Thu, Mar 20, 5:52 AM
Harbormaster completed remote builds in B33794: Diff 47498.Thu, Mar 20, 5:55 AM
kamil updated this revision to Diff 47504.Thu, Mar 20, 5:59 AM

fix typos

Harbormaster completed remote builds in B33800: Diff 47504.Thu, Mar 20, 6:17 AM
Closed by commit rCOMM993689cd4a3d: [web][keyserver] log Olm memory usage changes (authored by kamil). · Explain WhyThu, Mar 20, 6:21 AM
This revision was automatically updated to reflect the committed changes.
kamil added a commit: rCOMM993689cd4a3d: [web][keyserver] log Olm memory usage changes.

Revision Contents
Changeset List

PathSize
keyserver/
src/
cron/
10 lines
lib/
utils/
38 lines
web/
shared-worker/
worker/
3 lines

Diff 47498

View Options

keyserver/src/cron/cron.js

Show First 20 Lines • Show All 65 Lines • ▼ Show 20 Lines case State::DATA_HASH: {
} }
response->set_backupid(this->backupID); response->set_backupid(this->backupID);
this->holder = tools::generateHolder(this->dataHash, this->backupID); this->holder = tools::generateHolder(this->dataHash, this->backupID);
put_client_initialize_cxx(this->holder.c_str()); put_client_initialize_cxx(this->holder.c_str());
put_client_write_cxx( put_client_write_cxx(
this->holder.c_str(), this->holder.c_str(),
tools::getBlobPutField(blob::PutRequest::DataCase::kHolder), tools::getBlobPutField(blob::PutRequest::DataCase::kHolder),
this->holder.c_str()); this->holder.c_str());
put_client_blocking_read_cxx( put_client_blocking_read_cxx(this->holder.c_str());
this->holder.c_str()); // todo this should be avoided
// (blocking); we should be able to
// ignore responses; we probably want to
// delegate performing ops to separate
// threads in the base reactors
put_client_write_cxx( put_client_write_cxx(
this->holder.c_str(), this->holder.c_str(),
tools::getBlobPutField(blob::PutRequest::DataCase::kBlobHash), tools::getBlobPutField(blob::PutRequest::DataCase::kBlobHash),
this->dataHash.c_str()); this->dataHash.c_str());
rust::String responseStr = put_client_blocking_read_cxx( rust::String responseStr =
this->holder.c_str()); // todo this should be avoided put_client_blocking_read_cxx(this->holder.c_str());
// (blocking); we should be able to
// ignore responses; we probably
// want to delegate performing ops
// to separate threads in the base
// reactors
// data exists? // data exists?
if ((bool)tools::charPtrToInt(responseStr.c_str())) { if ((bool)tools::charPtrToInt(responseStr.c_str())) {
return std::make_unique<ServerBidiReactorStatus>( return std::make_unique<ServerBidiReactorStatus>(
grpc::Status::OK, true); grpc::Status::OK, true);
} }
return nullptr; return nullptr;
} }
case State::DATA_CHUNKS: { case State::DATA_CHUNKS: {
if (request.mutable_newcompactionchunk()->empty()) { if (request.mutable_newcompactionchunk()->empty()) {
return std::make_unique<ServerBidiReactorStatus>(grpc::Status::OK); return std::make_unique<ServerBidiReactorStatus>(grpc::Status::OK);
} }
put_client_write_cxx( put_client_write_cxx(
this->holder.c_str(), this->holder.c_str(),
tools::getBlobPutField(blob::PutRequest::DataCase::kDataChunk), tools::getBlobPutField(blob::PutRequest::DataCase::kDataChunk),
std::string(std::move(*request.mutable_newcompactionchunk())) std::string(std::move(*request.mutable_newcompactionchunk()))
.c_str()); .c_str());
put_client_blocking_read_cxx( put_client_blocking_read_cxx(this->holder.c_str());
this->holder.c_str()); // todo this should be avoided
// (blocking); we should be able to
// ignore responses; we probably want to
// delegate performing ops to separate
// threads in the base reactors
return nullptr; return nullptr;
} }
} }
throw std::runtime_error("new backup - invalid state"); throw std::runtime_error("new backup - invalid state");
} }
void CreateNewBackupReactor::terminateCallback() { void CreateNewBackupReactor::terminateCallback() {
Show All 18 Lines
View Options

lib/utils/olm-memory-utils.js

Show First 20 Lines • Show All 65 Lines • ▼ Show 20 Lines case State::DATA_HASH: {
} }
response->set_backupid(this->backupID); response->set_backupid(this->backupID);
this->holder = tools::generateHolder(this->dataHash, this->backupID); this->holder = tools::generateHolder(this->dataHash, this->backupID);
put_client_initialize_cxx(this->holder.c_str()); put_client_initialize_cxx(this->holder.c_str());
put_client_write_cxx( put_client_write_cxx(
this->holder.c_str(), this->holder.c_str(),
tools::getBlobPutField(blob::PutRequest::DataCase::kHolder), tools::getBlobPutField(blob::PutRequest::DataCase::kHolder),
this->holder.c_str()); this->holder.c_str());
put_client_blocking_read_cxx( put_client_blocking_read_cxx(this->holder.c_str());
this->holder.c_str()); // todo this should be avoided
// (blocking); we should be able to
// ignore responses; we probably want to
// delegate performing ops to separate
// threads in the base reactors
put_client_write_cxx( put_client_write_cxx(
this->holder.c_str(), this->holder.c_str(),
tools::getBlobPutField(blob::PutRequest::DataCase::kBlobHash), tools::getBlobPutField(blob::PutRequest::DataCase::kBlobHash),
this->dataHash.c_str()); this->dataHash.c_str());
rust::String responseStr = put_client_blocking_read_cxx( rust::String responseStr =
this->holder.c_str()); // todo this should be avoided put_client_blocking_read_cxx(this->holder.c_str());
// (blocking); we should be able to
// ignore responses; we probably
// want to delegate performing ops
// to separate threads in the base
// reactors
// data exists? // data exists?
if ((bool)tools::charPtrToInt(responseStr.c_str())) { if ((bool)tools::charPtrToInt(responseStr.c_str())) {
return std::make_unique<ServerBidiReactorStatus>( return std::make_unique<ServerBidiReactorStatus>(
grpc::Status::OK, true); grpc::Status::OK, true);
} }
return nullptr; return nullptr;
} }
case State::DATA_CHUNKS: { case State::DATA_CHUNKS: {
if (request.mutable_newcompactionchunk()->empty()) { if (request.mutable_newcompactionchunk()->empty()) {
return std::make_unique<ServerBidiReactorStatus>(grpc::Status::OK); return std::make_unique<ServerBidiReactorStatus>(grpc::Status::OK);
} }
put_client_write_cxx( put_client_write_cxx(
this->holder.c_str(), this->holder.c_str(),
tools::getBlobPutField(blob::PutRequest::DataCase::kDataChunk), tools::getBlobPutField(blob::PutRequest::DataCase::kDataChunk),
std::string(std::move(*request.mutable_newcompactionchunk())) std::string(std::move(*request.mutable_newcompactionchunk()))
.c_str()); .c_str());
put_client_blocking_read_cxx( put_client_blocking_read_cxx(this->holder.c_str());
this->holder.c_str()); // todo this should be avoided
// (blocking); we should be able to
// ignore responses; we probably want to
// delegate performing ops to separate
// threads in the base reactors
return nullptr; return nullptr;
} }
} }
throw std::runtime_error("new backup - invalid state"); throw std::runtime_error("new backup - invalid state");
} }
void CreateNewBackupReactor::terminateCallback() { void CreateNewBackupReactor::terminateCallback() {
Show All 18 Lines
View Options

web/shared-worker/worker/worker-crypto.js

Show First 20 Lines • Show All 65 Lines • ▼ Show 20 Lines case State::DATA_HASH: {
} }
response->set_backupid(this->backupID); response->set_backupid(this->backupID);
this->holder = tools::generateHolder(this->dataHash, this->backupID); this->holder = tools::generateHolder(this->dataHash, this->backupID);
put_client_initialize_cxx(this->holder.c_str()); put_client_initialize_cxx(this->holder.c_str());
put_client_write_cxx( put_client_write_cxx(
this->holder.c_str(), this->holder.c_str(),
tools::getBlobPutField(blob::PutRequest::DataCase::kHolder), tools::getBlobPutField(blob::PutRequest::DataCase::kHolder),
this->holder.c_str()); this->holder.c_str());
put_client_blocking_read_cxx( put_client_blocking_read_cxx(this->holder.c_str());
this->holder.c_str()); // todo this should be avoided
// (blocking); we should be able to
// ignore responses; we probably want to
// delegate performing ops to separate
// threads in the base reactors
put_client_write_cxx( put_client_write_cxx(
this->holder.c_str(), this->holder.c_str(),
tools::getBlobPutField(blob::PutRequest::DataCase::kBlobHash), tools::getBlobPutField(blob::PutRequest::DataCase::kBlobHash),
this->dataHash.c_str()); this->dataHash.c_str());
rust::String responseStr = put_client_blocking_read_cxx( rust::String responseStr =
this->holder.c_str()); // todo this should be avoided put_client_blocking_read_cxx(this->holder.c_str());
// (blocking); we should be able to
// ignore responses; we probably
// want to delegate performing ops
// to separate threads in the base
// reactors
// data exists? // data exists?
if ((bool)tools::charPtrToInt(responseStr.c_str())) { if ((bool)tools::charPtrToInt(responseStr.c_str())) {
return std::make_unique<ServerBidiReactorStatus>( return std::make_unique<ServerBidiReactorStatus>(
grpc::Status::OK, true); grpc::Status::OK, true);
} }
return nullptr; return nullptr;
} }
case State::DATA_CHUNKS: { case State::DATA_CHUNKS: {
if (request.mutable_newcompactionchunk()->empty()) { if (request.mutable_newcompactionchunk()->empty()) {
return std::make_unique<ServerBidiReactorStatus>(grpc::Status::OK); return std::make_unique<ServerBidiReactorStatus>(grpc::Status::OK);
} }
put_client_write_cxx( put_client_write_cxx(
this->holder.c_str(), this->holder.c_str(),
tools::getBlobPutField(blob::PutRequest::DataCase::kDataChunk), tools::getBlobPutField(blob::PutRequest::DataCase::kDataChunk),
std::string(std::move(*request.mutable_newcompactionchunk())) std::string(std::move(*request.mutable_newcompactionchunk()))
.c_str()); .c_str());
put_client_blocking_read_cxx( put_client_blocking_read_cxx(this->holder.c_str());
this->holder.c_str()); // todo this should be avoided
// (blocking); we should be able to
// ignore responses; we probably want to
// delegate performing ops to separate
// threads in the base reactors
return nullptr; return nullptr;
} }
} }
throw std::runtime_error("new backup - invalid state"); throw std::runtime_error("new backup - invalid state");
} }
void CreateNewBackupReactor::terminateCallback() { void CreateNewBackupReactor::terminateCallback() {
Show All 18 Lines