Page MenuHomePhabricator

[services] Rust Integration - Backup - Rust - Multiple put clients
ClosedPublic

Authored by karol on Sep 2 2022, 3:52 AM.
Tags
None
Referenced Files
F1748035: D5035.id16742.diff
Mon, May 13, 5:21 PM
F1748033: D5035.id16736.diff
Mon, May 13, 5:21 PM
F1746488: D5035.id16685.diff
Mon, May 13, 2:55 PM
F1746487: D5035.id16652.diff
Mon, May 13, 2:54 PM
F1746486: D5035.id16367.diff
Mon, May 13, 2:54 PM
F1746484: D5035.id16234.diff
Mon, May 13, 2:54 PM
F1746483: D5035.id.diff
Mon, May 13, 2:54 PM
Unknown Object (File)
Fri, May 10, 1:55 PM

Details

Summary

Depends on D5034

In order to be able to handle multiple connections at the same time, we have to store a collection of clients in the rust state instead of having just one.

Here, I'm adding the collection to the put blob client.

Test Plan

The backup service will not build successfully because of invalid calls from the c++ but I decided to divide the code so the diffs are not too big. This is fixed in the next diff in the stack - D5036

This will work:

cd services/backup/blob_client
cargo check

Diff Detail

Repository
rCOMM Comm
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Harbormaster returned this revision to the author for changes because remote builds failed.Sep 2 2022, 3:55 AM
Harbormaster failed remote builds in B11829: Diff 16234!
karol edited the summary of this revision. (Show Details)
karol edited the test plan for this revision. (Show Details)
karol added reviewers: tomek, varun, max.

Backup doesn't build but this is explained in the test plan section.

tomek requested changes to this revision.Sep 5 2022, 7:58 AM

Overall this might work but it seems like having a pool of clients and a queue with tasks could be a better solution.

services/backup/blob_client/src/put_client.rs
41 ↗(On Diff #16234)

This probably could be made more efficient by using RwLock

102–108 ↗(On Diff #16234)

We should format this code in the diff that introduced it.

305 ↗(On Diff #16234)

Can we make this message more verbose instead of adding just numbers?

This revision now requires changes to proceed.Sep 5 2022, 7:58 AM
services/backup/blob_client/src/put_client.rs
41 ↗(On Diff #16234)

I don't know, we probably should discuss that, mutex seems to be the easiest solution. I think this needs more research/discussion.

https://linear.app/comm/issue/ENG-1741/consider-using-rwlock-instead-of-mutex

102–108 ↗(On Diff #16234)

you're right

305 ↗(On Diff #16234)

sure

tomek requested changes to this revision.Sep 7 2022, 6:10 AM

I don't think an approach where every function takes the same parameter is a maintainable one. If all the functions share the same context, we should probably make this context a field of a struct and implement all the functions as methods of this struct. Is it possible to do something like that in cxx?

services/backup/blob_client/src/put_client.rs
49–54 ↗(On Diff #16367)

Can we keep the approach with match?

140–143 ↗(On Diff #16367)

Please avoid formatting the code that was introduced earlier

This revision now requires changes to proceed.Sep 7 2022, 6:10 AM
services/backup/blob_client/src/put_client.rs
49–54 ↗(On Diff #16367)

I don't mind

140–143 ↗(On Diff #16367)

I'll try. As this code's landed already, I gues I'm going to have to let this one go.

tomek requested changes to this revision.Sep 9 2022, 4:34 AM
tomek added inline comments.
services/backup/blob_client/src/lib.rs
18–20 ↗(On Diff #16496)

Is there a good reason for using *const c_char? Are there any other CXX types that represent a string and don't require using raw pointers?

services/backup/blob_client/src/put_client.rs
47 ↗(On Diff #16496)

Is there a downside of using &str?

This revision now requires changes to proceed.Sep 9 2022, 4:34 AM
services/backup/blob_client/src/lib.rs
18–20 ↗(On Diff #16496)

In all tutorials they use const char* but you made me think and I don't see why rust::String couldn't work. It is derived in the cxx bridge header file, so it is accessible and can be constructed from the c++ std::string. I've not tried it though.

I'm going to give it a try, thanks.

services/backup/blob_client/src/put_client.rs
47 ↗(On Diff #16496)

And is there one of using &String? I think it's better to borrow a heap-allocated string instead of converting it into a str, right?

karol added inline comments.
services/backup/blob_client/src/lib.rs
18–20 ↗(On Diff #16496)
services/backup/blob_client/src/put_client.rs
47 ↗(On Diff #16496)

Could also use rust::Str https://cxx.rs/binding/str.html, and then this would be &str on the rust side which is more appropriate.

tomek requested changes to this revision.Sep 13 2022, 7:16 AM

The CI reported some compilation issues

services/backup/blob_client/src/lib.rs
18–20 ↗(On Diff #16496)

It would be a lot better to update this diff instead of creating a followup

services/backup/blob_client/src/put_client.rs
40 ↗(On Diff #16496)

Can we update this diff to not use Arc?

47 ↗(On Diff #16496)

And is there one of using &String?

Yet, it is
https://blog.logrocket.com/understanding-rust-string-str/

For example, since &String can be coerced to &str, but not the other way around, it usually makes sense to use &str as a parameter type, if you just need a read-only view of a string.

So using &str in every place where it can be used is a good practice because it makes the API easier to use. On the other hand

Further, if a function needs to mutate a given String, there is no point in passing in a &str, since it will be immutable and you would have to create a String from it and return it afterwards. If mutation is what you need, &mut String is the way to go.

So if the function mutates, then String is a good idea.

But in all the cases &str is strongly preferred, unless it can't be used.

This revision now requires changes to proceed.Sep 13 2022, 7:16 AM
services/backup/blob_client/src/put_client.rs
40 ↗(On Diff #16496)

https://phab.comm.dev/D5114 was merged, should just need to rebase now

services/backup/blob_client/src/put_client.rs
47 ↗(On Diff #16496)

Ok, fair enough, will change to &str then.

would really like for this to be rebased, the Arc usage for the other lazy_static items has been removed on master, and would like to avoid potentially adding them back.

This revision now requires changes to proceed.Sep 14 2022, 10:48 AM

Btw, if that was the only concern I think you could just accept it, rebasing to master was pretty easy and straightforward and would enable landing some diffs already reducing cycles.

This revision is now accepted and ready to land.Sep 15 2022, 4:23 PM
This revision was landed with ongoing or failed builds.Sep 16 2022, 1:54 AM
This revision was automatically updated to reflect the committed changes.

Not sure what happened here, but according to CI

This revision was landed with ongoing or failed builds.

We should (almost) never land anything with failed builds!