Page MenuHomePhabricator
Differential D4942

[DRAFT] [services] Backup - Connect to Blob - Prepare for bidi stream
AbandonedPublic
Actions

Authored by • karol on Aug 24 2022, 5:56 AM.
Tags
None
Referenced Files
Unknown Object (File)
Tue, Mar 25, 4:52 PM
Unknown Object (File)
Wed, Mar 19, 12:53 PM
Unknown Object (File)
Mon, Mar 17, 9:40 PM
Unknown Object (File)
Sat, Mar 15, 9:12 PM
Unknown Object (File)
Tue, Mar 4, 9:37 AM
Unknown Object (File)
Tue, Mar 4, 9:37 AM
Unknown Object (File)
Tue, Mar 4, 9:36 AM
Unknown Object (File)
Tue, Mar 4, 9:29 AM
View All Files
Subscribers
• abosh
ashoat
atul
tomek

Details

Reviewers
tomek
varun
• max
• jon
Summary

Depends on D4889

Prepare the put client to handle bidirectional stream operations.

Test Plan

Test plan in D4946

Diff Detail

Repository
rCOMM Comm
Lint
No Lint Coverage
Unit
No Test Coverage

Event Timeline

• karol created this revision.Aug 24 2022, 5:56 AM
Herald added subscribers: • abosh, atul, tomek, ashoat. · View Herald TranscriptAug 24 2022, 5:56 AM
• karol retitled this revision from [services] Backup - Connect to Blob - Prepare for bidi stream to [DRAFT] [services] Backup - Connect to Blob - Prepare for bidi stream.Aug 24 2022, 6:00 AM
• karol edited the summary of this revision. (Show Details)
• karol edited the test plan for this revision. (Show Details)
• karol added reviewers: tomek, varun, • max.
• karol added a parent revision: D4889: [services] Backup - Connect to Blob - Compile protos.
• karol added a child revision: D4943: [DRAFT] [services] Blob - check for holder/hash presence in terminate callback.
Harbormaster completed remote builds in B11578: Diff 15919.Aug 24 2022, 6:06 AM
• karol requested review of this revision.Aug 24 2022, 6:06 AM
tomek requested changes to this revision.Aug 29 2022, 4:24 AM
tomek added inline comments.
services/backup/blob_client/src/put_client.rs
1 ↗(On Diff #15919)

What is the reason for this change?

22–25 ↗(On Diff #15919)

These names are slightly confusing. Can we chose one approach and stick to it? (either tx and rx or transmitter and receiver)

72–76 ↗(On Diff #15919)

Is there a way to lock just once and set these four values?

82–91 ↗(On Diff #15919)

Could you explain what's going on here? It seems like we take a rx, read from it and assign it again. Why this assignment is needed?

This revision now requires changes to proceed.Aug 29 2022, 4:24 AM
Herald added a reviewer: • jon. · View Herald TranscriptAug 29 2022, 4:24 AM
• karol added inline comments.Aug 29 2022, 5:50 AM
services/backup/blob_client/src/put_client.rs
1 ↗(On Diff #15919)

A leftover, it will be reverted, thx.

22–25 ↗(On Diff #15919)

Sure, no problem.

72–76 ↗(On Diff #15919)

It's fixed in D4974

82–91 ↗(On Diff #15919)

It's a hack, I had to do this, because when I wanted to use rx without take, rust would complain that I couldn't use it as mut (IIRC because it's a struct member).

• karol updated this revision to Diff 16071.Aug 29 2022, 7:06 AM

address comments

Harbormaster completed remote builds in B11685: Diff 16071.Aug 29 2022, 7:16 AM
• jon added inline comments.Aug 29 2022, 9:56 AM
services/backup/blob_client/src/put_client.rs
12–16

If having a sender and receiver is intrinsic (e.g. the existence of it requires these things to be true) to a bidiclient, then I don't think we should have option on each field. Instead we should use Option<BidiClient> to see if it's been initialized. If you did get Some(BidiClient), then you can assume that it has valid sender/receiver fields.

20–26
30–43

Little more concise way to express this, and avoids acquiring a lock twice

If you do the Option<BidiClient> comment I did above, then you would just write:

fn is_initialized() -> bool {
  CLIENT.lock().expect("access client").is_some()
}
67

We should be using a logging utilities for this. Then we can control the noise level with RUST_LOG

72

would be nice for the expect to be a bit more descriptive. If we do see failures, then just "access client" being copy pasted every doesn't help too much

• karol added a comment.EditedAug 30 2022, 2:14 AM

@jon if you have that many comments, please, request changes.

services/backup/blob_client/src/put_client.rs
12–16

I initially wanted to do it like this but it didn't work for some reason. I'll try to do this but wouldn't give it a high prio. https://linear.app/comm/issue/ENG-1716/try-using-option-higher-in-hierarchy

20–26

https://linear.app/comm/issue/ENG-1716/try-using-option-higher-in-hierarchy

30–43

fixed in D4974

67

I couldn't make it work, they simply will not show on the screen. https://linear.app/comm/issue/ENG-1717/use-tracing-utilities-instead-of-println

72

I think in the error message there is a line number so that's helpful enough. Also, we're getting rid of expects and unwraps later in the stack.

• karol planned changes to this revision.Aug 30 2022, 2:44 AM
• karol removed a child revision: D4943: [DRAFT] [services] Blob - check for holder/hash presence in terminate callback.Sep 1 2022, 3:40 AM
• karol abandoned this revision.Sep 1 2022, 4:30 AM

Abandoning this stack in favor of the new one that starts @ D5002.

All the comments have been either addressed inline or in the new stack.

Revision Contents
Changeset List

PathSize
services/
backup/
blob_client/
src/
1 line
2 lines
132 lines

Diff 16071

View Options

services/backup/blob_client/src/constants.rs

Show First 20 Lines • Show All 130 Lines • ▼ Show 20 Lines SQL`
CREATE TABLE sessions ( CREATE TABLE sessions (
id bigint(20) NOT NULL, id bigint(20) NOT NULL,
user bigint(20) NOT NULL, user bigint(20) NOT NULL,
cookie bigint(20) NOT NULL, cookie bigint(20) NOT NULL,
query json NOT NULL, query json NOT NULL,
creation_time bigint(20) NOT NULL, creation_time bigint(20) NOT NULL,
last_update bigint(20) NOT NULL, last_update bigint(20) NOT NULL,
last_validated bigint(20) NOT NULL last_validated bigint(20) NOT NULL,
public_key char(116) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1; ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE threads ( CREATE TABLE threads (
id bigint(20) NOT NULL, id bigint(20) NOT NULL,
type tinyint(3) NOT NULL, type tinyint(3) NOT NULL,
name varchar(191) COLLATE utf8mb4_bin DEFAULT NULL, name varchar(191) COLLATE utf8mb4_bin DEFAULT NULL,
description mediumtext COLLATE utf8mb4_bin, description mediumtext COLLATE utf8mb4_bin,
parent_thread_id bigint(20) DEFAULT NULL, parent_thread_id bigint(20) DEFAULT NULL,
Show All 30 Lines SQL`
secret varchar(255) NOT NULL, secret varchar(255) NOT NULL,
creation_time bigint(20) NOT NULL, creation_time bigint(20) NOT NULL,
extra json DEFAULT NULL extra json DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8; ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE users ( CREATE TABLE users (
id bigint(20) NOT NULL, id bigint(20) NOT NULL,
username varchar(${usernameMaxLength}) COLLATE utf8mb4_bin NOT NULL, username varchar(${usernameMaxLength}) COLLATE utf8mb4_bin NOT NULL,
hash char(60) COLLATE utf8mb4_bin DEFAULT NULL, hash char(60) COLLATE utf8mb4_bin NOT NULL,
avatar varchar(191) COLLATE utf8mb4_bin DEFAULT NULL, avatar varchar(191) COLLATE utf8mb4_bin DEFAULT NULL,
creation_time bigint(20) NOT NULL creation_time bigint(20) NOT NULL,
public_key char(116) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;
CREATE TABLE relationships_undirected ( CREATE TABLE relationships_undirected (
user1 bigint(20) NOT NULL, user1 bigint(20) NOT NULL,
user2 bigint(20) NOT NULL, user2 bigint(20) NOT NULL,
status tinyint(1) UNSIGNED NOT NULL status tinyint(1) UNSIGNED NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1; ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
▲ Show 20 Lines • Show All 85 Lines • ▼ Show 20 Lines SQL`
ADD KEY entry (entry); ADD KEY entry (entry);
ALTER TABLE roles ALTER TABLE roles
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD KEY thread (thread); ADD KEY thread (thread);
ALTER TABLE sessions ALTER TABLE sessions
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD KEY user (user); ADD KEY user (user),
ADD UNIQUE INDEX public_key (public_key);
ALTER TABLE threads ALTER TABLE threads
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD INDEX parent_thread_id (parent_thread_id), ADD INDEX parent_thread_id (parent_thread_id),
ADD INDEX containing_thread_id (containing_thread_id), ADD INDEX containing_thread_id (containing_thread_id),
ADD INDEX community (community); ADD INDEX community (community);
ALTER TABLE updates ALTER TABLE updates
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD INDEX user_time (user,time), ADD INDEX user_time (user,time),
ADD INDEX target_time (target, time), ADD INDEX target_time (target, time),
ADD INDEX user_key_target_type_time (user, \`key\`, target, type, time), ADD INDEX user_key_target_type_time (user, \`key\`, target, type, time),
ADD INDEX user_key_type_time (user, \`key\`, type, time), ADD INDEX user_key_type_time (user, \`key\`, type, time),
ADD INDEX user_key_time (user, \`key\`, time); ADD INDEX user_key_time (user, \`key\`, time);
ALTER TABLE uploads ALTER TABLE uploads
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD INDEX container (container); ADD INDEX container (container);
ALTER TABLE users ALTER TABLE users
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD UNIQUE KEY username (username); ADD UNIQUE KEY username (username),
ADD UNIQUE INDEX public_key (public_key);
ALTER TABLE relationships_undirected ALTER TABLE relationships_undirected
ADD UNIQUE KEY user1_user2 (user1,user2), ADD UNIQUE KEY user1_user2 (user1,user2),
ADD UNIQUE KEY user2_user1 (user2,user1); ADD UNIQUE KEY user2_user1 (user2,user1);
ALTER TABLE relationships_directed ALTER TABLE relationships_directed
ADD UNIQUE KEY user1_user2 (user1,user2), ADD UNIQUE KEY user1_user2 (user1,user2),
ADD UNIQUE KEY user2_user1 (user2,user1); ADD UNIQUE KEY user2_user1 (user2,user1);
▲ Show 20 Lines • Show All 87 Lines • Show Last 20 Lines
View Options

services/backup/blob_client/src/lib.rs

Show First 20 Lines • Show All 130 Lines • ▼ Show 20 Lines SQL`
CREATE TABLE sessions ( CREATE TABLE sessions (
id bigint(20) NOT NULL, id bigint(20) NOT NULL,
user bigint(20) NOT NULL, user bigint(20) NOT NULL,
cookie bigint(20) NOT NULL, cookie bigint(20) NOT NULL,
query json NOT NULL, query json NOT NULL,
creation_time bigint(20) NOT NULL, creation_time bigint(20) NOT NULL,
last_update bigint(20) NOT NULL, last_update bigint(20) NOT NULL,
last_validated bigint(20) NOT NULL last_validated bigint(20) NOT NULL,
public_key char(116) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1; ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE threads ( CREATE TABLE threads (
id bigint(20) NOT NULL, id bigint(20) NOT NULL,
type tinyint(3) NOT NULL, type tinyint(3) NOT NULL,
name varchar(191) COLLATE utf8mb4_bin DEFAULT NULL, name varchar(191) COLLATE utf8mb4_bin DEFAULT NULL,
description mediumtext COLLATE utf8mb4_bin, description mediumtext COLLATE utf8mb4_bin,
parent_thread_id bigint(20) DEFAULT NULL, parent_thread_id bigint(20) DEFAULT NULL,
Show All 30 Lines SQL`
secret varchar(255) NOT NULL, secret varchar(255) NOT NULL,
creation_time bigint(20) NOT NULL, creation_time bigint(20) NOT NULL,
extra json DEFAULT NULL extra json DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8; ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE users ( CREATE TABLE users (
id bigint(20) NOT NULL, id bigint(20) NOT NULL,
username varchar(${usernameMaxLength}) COLLATE utf8mb4_bin NOT NULL, username varchar(${usernameMaxLength}) COLLATE utf8mb4_bin NOT NULL,
hash char(60) COLLATE utf8mb4_bin DEFAULT NULL, hash char(60) COLLATE utf8mb4_bin NOT NULL,
avatar varchar(191) COLLATE utf8mb4_bin DEFAULT NULL, avatar varchar(191) COLLATE utf8mb4_bin DEFAULT NULL,
creation_time bigint(20) NOT NULL creation_time bigint(20) NOT NULL,
public_key char(116) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;
CREATE TABLE relationships_undirected ( CREATE TABLE relationships_undirected (
user1 bigint(20) NOT NULL, user1 bigint(20) NOT NULL,
user2 bigint(20) NOT NULL, user2 bigint(20) NOT NULL,
status tinyint(1) UNSIGNED NOT NULL status tinyint(1) UNSIGNED NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1; ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
▲ Show 20 Lines • Show All 85 Lines • ▼ Show 20 Lines SQL`
ADD KEY entry (entry); ADD KEY entry (entry);
ALTER TABLE roles ALTER TABLE roles
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD KEY thread (thread); ADD KEY thread (thread);
ALTER TABLE sessions ALTER TABLE sessions
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD KEY user (user); ADD KEY user (user),
ADD UNIQUE INDEX public_key (public_key);
ALTER TABLE threads ALTER TABLE threads
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD INDEX parent_thread_id (parent_thread_id), ADD INDEX parent_thread_id (parent_thread_id),
ADD INDEX containing_thread_id (containing_thread_id), ADD INDEX containing_thread_id (containing_thread_id),
ADD INDEX community (community); ADD INDEX community (community);
ALTER TABLE updates ALTER TABLE updates
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD INDEX user_time (user,time), ADD INDEX user_time (user,time),
ADD INDEX target_time (target, time), ADD INDEX target_time (target, time),
ADD INDEX user_key_target_type_time (user, \`key\`, target, type, time), ADD INDEX user_key_target_type_time (user, \`key\`, target, type, time),
ADD INDEX user_key_type_time (user, \`key\`, type, time), ADD INDEX user_key_type_time (user, \`key\`, type, time),
ADD INDEX user_key_time (user, \`key\`, time); ADD INDEX user_key_time (user, \`key\`, time);
ALTER TABLE uploads ALTER TABLE uploads
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD INDEX container (container); ADD INDEX container (container);
ALTER TABLE users ALTER TABLE users
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD UNIQUE KEY username (username); ADD UNIQUE KEY username (username),
ADD UNIQUE INDEX public_key (public_key);
ALTER TABLE relationships_undirected ALTER TABLE relationships_undirected
ADD UNIQUE KEY user1_user2 (user1,user2), ADD UNIQUE KEY user1_user2 (user1,user2),
ADD UNIQUE KEY user2_user1 (user2,user1); ADD UNIQUE KEY user2_user1 (user2,user1);
ALTER TABLE relationships_directed ALTER TABLE relationships_directed
ADD UNIQUE KEY user1_user2 (user1,user2), ADD UNIQUE KEY user1_user2 (user1,user2),
ADD UNIQUE KEY user2_user1 (user2,user1); ADD UNIQUE KEY user2_user1 (user2,user1);
▲ Show 20 Lines • Show All 87 Lines • Show Last 20 Lines
View Options

services/backup/blob_client/src/put_client.rs

Show First 20 Lines • Show All 130 Lines • ▼ Show 20 Lines SQL`
CREATE TABLE sessions ( CREATE TABLE sessions (
id bigint(20) NOT NULL, id bigint(20) NOT NULL,
user bigint(20) NOT NULL, user bigint(20) NOT NULL,
cookie bigint(20) NOT NULL, cookie bigint(20) NOT NULL,
query json NOT NULL, query json NOT NULL,
creation_time bigint(20) NOT NULL, creation_time bigint(20) NOT NULL,
last_update bigint(20) NOT NULL, last_update bigint(20) NOT NULL,
last_validated bigint(20) NOT NULL last_validated bigint(20) NOT NULL,
public_key char(116) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1; ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
CREATE TABLE threads ( CREATE TABLE threads (
id bigint(20) NOT NULL, id bigint(20) NOT NULL,
type tinyint(3) NOT NULL, type tinyint(3) NOT NULL,
name varchar(191) COLLATE utf8mb4_bin DEFAULT NULL, name varchar(191) COLLATE utf8mb4_bin DEFAULT NULL,
description mediumtext COLLATE utf8mb4_bin, description mediumtext COLLATE utf8mb4_bin,
parent_thread_id bigint(20) DEFAULT NULL, parent_thread_id bigint(20) DEFAULT NULL,
Show All 30 Lines SQL`
secret varchar(255) NOT NULL, secret varchar(255) NOT NULL,
creation_time bigint(20) NOT NULL, creation_time bigint(20) NOT NULL,
extra json DEFAULT NULL extra json DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8; ) ENGINE=InnoDB DEFAULT CHARSET=utf8;
CREATE TABLE users ( CREATE TABLE users (
id bigint(20) NOT NULL, id bigint(20) NOT NULL,
username varchar(${usernameMaxLength}) COLLATE utf8mb4_bin NOT NULL, username varchar(${usernameMaxLength}) COLLATE utf8mb4_bin NOT NULL,
hash char(60) COLLATE utf8mb4_bin DEFAULT NULL, hash char(60) COLLATE utf8mb4_bin NOT NULL,
avatar varchar(191) COLLATE utf8mb4_bin DEFAULT NULL, avatar varchar(191) COLLATE utf8mb4_bin DEFAULT NULL,
creation_time bigint(20) NOT NULL creation_time bigint(20) NOT NULL,
public_key char(116) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin; ) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin;
CREATE TABLE relationships_undirected ( CREATE TABLE relationships_undirected (
user1 bigint(20) NOT NULL, user1 bigint(20) NOT NULL,
user2 bigint(20) NOT NULL, user2 bigint(20) NOT NULL,
status tinyint(1) UNSIGNED NOT NULL status tinyint(1) UNSIGNED NOT NULL
) ENGINE=InnoDB DEFAULT CHARSET=latin1; ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
▲ Show 20 Lines • Show All 85 Lines • ▼ Show 20 Lines SQL`
ADD KEY entry (entry); ADD KEY entry (entry);
ALTER TABLE roles ALTER TABLE roles
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD KEY thread (thread); ADD KEY thread (thread);
ALTER TABLE sessions ALTER TABLE sessions
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD KEY user (user); ADD KEY user (user),
ADD UNIQUE INDEX public_key (public_key);
ALTER TABLE threads ALTER TABLE threads
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD INDEX parent_thread_id (parent_thread_id), ADD INDEX parent_thread_id (parent_thread_id),
ADD INDEX containing_thread_id (containing_thread_id), ADD INDEX containing_thread_id (containing_thread_id),
ADD INDEX community (community); ADD INDEX community (community);
ALTER TABLE updates ALTER TABLE updates
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD INDEX user_time (user,time), ADD INDEX user_time (user,time),
ADD INDEX target_time (target, time), ADD INDEX target_time (target, time),
ADD INDEX user_key_target_type_time (user, \`key\`, target, type, time), ADD INDEX user_key_target_type_time (user, \`key\`, target, type, time),
ADD INDEX user_key_type_time (user, \`key\`, type, time), ADD INDEX user_key_type_time (user, \`key\`, type, time),
ADD INDEX user_key_time (user, \`key\`, time); ADD INDEX user_key_time (user, \`key\`, time);
ALTER TABLE uploads ALTER TABLE uploads
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD INDEX container (container); ADD INDEX container (container);
ALTER TABLE users ALTER TABLE users
ADD PRIMARY KEY (id), ADD PRIMARY KEY (id),
ADD UNIQUE KEY username (username); ADD UNIQUE KEY username (username),
ADD UNIQUE INDEX public_key (public_key);
ALTER TABLE relationships_undirected ALTER TABLE relationships_undirected
ADD UNIQUE KEY user1_user2 (user1,user2), ADD UNIQUE KEY user1_user2 (user1,user2),
ADD UNIQUE KEY user2_user1 (user2,user1); ADD UNIQUE KEY user2_user1 (user2,user1);
ALTER TABLE relationships_directed ALTER TABLE relationships_directed
ADD UNIQUE KEY user1_user2 (user1,user2), ADD UNIQUE KEY user1_user2 (user1,user2),
ADD UNIQUE KEY user2_user1 (user2,user1); ADD UNIQUE KEY user2_user1 (user2,user1);
▲ Show 20 Lines • Show All 87 Lines • Show Last 20 Lines