Page MenuHomePhabricator
Differential D3399

services: Remove gRPC source after installing
ClosedPublic
Actions

Authored by • jim on Mar 10 2022, 11:36 AM.
Tags
None
Referenced Files
Unknown Object (File)
Wed, Apr 16, 10:34 AM
Unknown Object (File)
Wed, Apr 16, 8:55 AM
Unknown Object (File)
Wed, Apr 16, 8:10 AM
Unknown Object (File)
Wed, Apr 16, 5:08 AM
Unknown Object (File)
Wed, Apr 16, 12:10 AM
Unknown Object (File)
Mon, Apr 14, 10:48 PM
Unknown Object (File)
Tue, Apr 8, 6:25 PM
Unknown Object (File)
Tue, Apr 8, 5:31 PM
View All Files
Subscribers
• adrian
atul
• benschac
tomek

Details

Reviewers
• karol
ashoat
• max
Commits
rCOMM7f17a64e3155: services: Remove gRPC source after installing
Summary

This reduces size of the Docker base image from 3.03GB to 1.29GB.

Also:

services: Pass gRPC build flags at build time
services: Better Docker layering in base image
services: Remove unused gRPC build variable

Test Plan

Run npm run build-all and run-all

Diff Detail

Repository
rCOMM Comm
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

• jim created this revision.Mar 10 2022, 11:36 AM
Herald added subscribers: • benschac, atul, • adrian, tomek. · View Herald TranscriptMar 10 2022, 11:36 AM
Harbormaster completed remote builds in B7286: Diff 10274.Mar 10 2022, 12:17 PM
• jim requested review of this revision.Mar 10 2022, 12:17 PM
ashoat accepted this revision.Mar 10 2022, 8:05 PM
ashoat added inline comments.
services/base-image/contents/install_grpc.sh
30–31 ↗(On Diff #10274)

Thanks for adding this comment

This revision is now accepted and ready to land.Mar 10 2022, 8:05 PM
• karol added a comment.Mar 15 2022, 3:12 AM

What is the reason behind having grpc flags passed to the cmake in the bash script instead of having them directly in the cmake file?

services/backup/docker-server/contents/build_server.sh
5 ↗(On Diff #10274)

What are these changes for?
I think it's safer with /s

• jim requested review of this revision.Mar 15 2022, 7:27 AM
In D3399#92777, @karol-bisztyga wrote:

What is the reason behind having grpc flags passed to the cmake in the bash script instead of having them directly in the cmake file?

Because that is when gRPC is built. In the CMake file, it's just using the pre-installed gRPC library.

See comments here https://phabricator.ashoat.com/D3198?id=10038#inline-19455 and here https://linear.app/comm/issue/ENG-824#comment-6555f66b.

services/backup/docker-server/contents/build_server.sh
5 ↗(On Diff #10274)

The changes are needed because I changed the WORKDIR in the base image from /. And I agree that it's preferable to change to an absolute path in this script OR in the backup service Dockerfile do RUN cd /transferred/server && ../scripts/build-server.sh. As in, if this script requires itself to be run from a certain place, that place should be the target directory, not the root directory.

• karol added a comment.Mar 15 2022, 8:43 AM

Ah yes, sorry I get it now, it's alright!

services/backup/docker-server/contents/build_server.sh
5 ↗(On Diff #10274)

I just prefer an absolute path, it's a matter of design I believe. Either you make it more error-resistant in terms of where the script is being run from or in terms of where exactly is the target path in the file system.

ashoat accepted this revision.Mar 15 2022, 8:16 PM
This revision is now accepted and ready to land.Mar 15 2022, 8:16 PM
• jim updated this revision to Diff 10509.Mar 17 2022, 3:53 PM

Rebased

Harbormaster completed remote builds in B7460: Diff 10509.Mar 17 2022, 4:24 PM
Closed by commit rCOMM7f17a64e3155: services: Remove gRPC source after installing (authored by • jim). · Explain WhyMar 18 2022, 5:58 AM
This revision was automatically updated to reflect the committed changes.
• jim added a commit: rCOMM7f17a64e3155: services: Remove gRPC source after installing.

Revision Contents
Changeset List

PathSize
services/
backup/
docker-server/
contents/
4 lines
server/
1 line
cmake-components/
9 lines
base-image/
20 lines
contents/
33 lines
blob/
docker-server/
contents/
4 lines
server/
cmake-components/
9 lines
tunnelbroker/
9 lines
1 line

Diff 10515

View Options

services/backup/docker-server/contents/build_server.sh

#include <grpcpp/grpcpp.h> #include <grpcpp/grpcpp.h>
namespace comm { namespace comm {
namespace network { namespace network {
namespace reactor { namespace reactor {
template <class Request, class Response> template <class Request, class Response>
class ClientWriteReactorBase : public grpc::ClientWriteReactor<Request> { class ClientWriteReactorBase : public grpc::ClientWriteReactor<Request> {
grpc::Status status; grpc::Status status = grpc::Status::OK;
bool done = false; bool done = false;
bool initialized = 0; bool initialized = 0;
Request request; Request request;
public: public:
Response response; Response response;
grpc::ClientContext context; grpc::ClientContext context;
void nextWrite(); void nextWrite();
void OnWriteDone(bool ok) override; void OnWriteDone(bool ok) override;
void terminate(const grpc::Status &status); void terminate(const grpc::Status &status);
bool isDone(); bool isDone();
void OnDone(const grpc::Status &status) override; void OnDone(const grpc::Status &status) override;
virtual std::unique_ptr<grpc::Status> prepareRequest(Request &request) = 0; virtual std::unique_ptr<grpc::Status> prepareRequest(Request &request) = 0;
virtual void validate(){};
virtual void doneCallback(){}; virtual void doneCallback(){};
virtual void terminateCallback(){}; virtual void terminateCallback(){};
}; };
template <class Request, class Response> template <class Request, class Response>
void ClientWriteReactorBase<Request, Response>::nextWrite() { void ClientWriteReactorBase<Request, Response>::nextWrite() {
this->request = Request(); this->request = Request();
std::unique_ptr<grpc::Status> status = this->prepareRequest(this->request); std::unique_ptr<grpc::Status> status = this->prepareRequest(this->request);
Show All 15 Lines if (!ok) {
return; return;
} }
this->nextWrite(); this->nextWrite();
} }
template <class Request, class Response> template <class Request, class Response>
void ClientWriteReactorBase<Request, Response>::terminate( void ClientWriteReactorBase<Request, Response>::terminate(
const grpc::Status &status) { const grpc::Status &status) {
this->terminateCallback(); if (this->status.ok()) {
if (this->done) { this->status = status;
return;
} }
if (!this->status.ok()) { if (!this->status.ok()) {
std::cout << "error: " << this->status.error_message() << std::endl; std::cout << "error: " << this->status.error_message() << std::endl;
} }
this->status = status; if (this->done) {
return;
}
this->terminateCallback();
try {
this->validate();
} catch (std::runtime_error &e) {
this->status = grpc::Status(grpc::StatusCode::INTERNAL, e.what());
}
this->done = true; this->done = true;
this->StartWritesDone(); this->StartWritesDone();
} }
template <class Request, class Response> template <class Request, class Response>
bool ClientWriteReactorBase<Request, Response>::isDone() { bool ClientWriteReactorBase<Request, Response>::isDone() {
return this->done; return this->done;
} }
Show All 11 Lines
View Options

services/backup/docker-server/contents/server/CMakeLists.txt

#include <grpcpp/grpcpp.h> #include <grpcpp/grpcpp.h>
namespace comm { namespace comm {
namespace network { namespace network {
namespace reactor { namespace reactor {
template <class Request, class Response> template <class Request, class Response>
class ClientWriteReactorBase : public grpc::ClientWriteReactor<Request> { class ClientWriteReactorBase : public grpc::ClientWriteReactor<Request> {
grpc::Status status; grpc::Status status = grpc::Status::OK;
bool done = false; bool done = false;
bool initialized = 0; bool initialized = 0;
Request request; Request request;
public: public:
Response response; Response response;
grpc::ClientContext context; grpc::ClientContext context;
void nextWrite(); void nextWrite();
void OnWriteDone(bool ok) override; void OnWriteDone(bool ok) override;
void terminate(const grpc::Status &status); void terminate(const grpc::Status &status);
bool isDone(); bool isDone();
void OnDone(const grpc::Status &status) override; void OnDone(const grpc::Status &status) override;
virtual std::unique_ptr<grpc::Status> prepareRequest(Request &request) = 0; virtual std::unique_ptr<grpc::Status> prepareRequest(Request &request) = 0;
virtual void validate(){};
virtual void doneCallback(){}; virtual void doneCallback(){};
virtual void terminateCallback(){}; virtual void terminateCallback(){};
}; };
template <class Request, class Response> template <class Request, class Response>
void ClientWriteReactorBase<Request, Response>::nextWrite() { void ClientWriteReactorBase<Request, Response>::nextWrite() {
this->request = Request(); this->request = Request();
std::unique_ptr<grpc::Status> status = this->prepareRequest(this->request); std::unique_ptr<grpc::Status> status = this->prepareRequest(this->request);
Show All 15 Lines if (!ok) {
return; return;
} }
this->nextWrite(); this->nextWrite();
} }
template <class Request, class Response> template <class Request, class Response>
void ClientWriteReactorBase<Request, Response>::terminate( void ClientWriteReactorBase<Request, Response>::terminate(
const grpc::Status &status) { const grpc::Status &status) {
this->terminateCallback(); if (this->status.ok()) {
if (this->done) { this->status = status;
return;
} }
if (!this->status.ok()) { if (!this->status.ok()) {
std::cout << "error: " << this->status.error_message() << std::endl; std::cout << "error: " << this->status.error_message() << std::endl;
} }
this->status = status; if (this->done) {
return;
}
this->terminateCallback();
try {
this->validate();
} catch (std::runtime_error &e) {
this->status = grpc::Status(grpc::StatusCode::INTERNAL, e.what());
}
this->done = true; this->done = true;
this->StartWritesDone(); this->StartWritesDone();
} }
template <class Request, class Response> template <class Request, class Response>
bool ClientWriteReactorBase<Request, Response>::isDone() { bool ClientWriteReactorBase<Request, Response>::isDone() {
return this->done; return this->done;
} }
Show All 11 Lines
View Options

services/backup/docker-server/contents/server/cmake-components/grpc.cmake

#include <grpcpp/grpcpp.h> #include <grpcpp/grpcpp.h>
namespace comm { namespace comm {
namespace network { namespace network {
namespace reactor { namespace reactor {
template <class Request, class Response> template <class Request, class Response>
class ClientWriteReactorBase : public grpc::ClientWriteReactor<Request> { class ClientWriteReactorBase : public grpc::ClientWriteReactor<Request> {
grpc::Status status; grpc::Status status = grpc::Status::OK;
bool done = false; bool done = false;
bool initialized = 0; bool initialized = 0;
Request request; Request request;
public: public:
Response response; Response response;
grpc::ClientContext context; grpc::ClientContext context;
void nextWrite(); void nextWrite();
void OnWriteDone(bool ok) override; void OnWriteDone(bool ok) override;
void terminate(const grpc::Status &status); void terminate(const grpc::Status &status);
bool isDone(); bool isDone();
void OnDone(const grpc::Status &status) override; void OnDone(const grpc::Status &status) override;
virtual std::unique_ptr<grpc::Status> prepareRequest(Request &request) = 0; virtual std::unique_ptr<grpc::Status> prepareRequest(Request &request) = 0;
virtual void validate(){};
virtual void doneCallback(){}; virtual void doneCallback(){};
virtual void terminateCallback(){}; virtual void terminateCallback(){};
}; };
template <class Request, class Response> template <class Request, class Response>
void ClientWriteReactorBase<Request, Response>::nextWrite() { void ClientWriteReactorBase<Request, Response>::nextWrite() {
this->request = Request(); this->request = Request();
std::unique_ptr<grpc::Status> status = this->prepareRequest(this->request); std::unique_ptr<grpc::Status> status = this->prepareRequest(this->request);
Show All 15 Lines if (!ok) {
return; return;
} }
this->nextWrite(); this->nextWrite();
} }
template <class Request, class Response> template <class Request, class Response>
void ClientWriteReactorBase<Request, Response>::terminate( void ClientWriteReactorBase<Request, Response>::terminate(
const grpc::Status &status) { const grpc::Status &status) {
this->terminateCallback(); if (this->status.ok()) {
if (this->done) { this->status = status;
return;
} }
if (!this->status.ok()) { if (!this->status.ok()) {
std::cout << "error: " << this->status.error_message() << std::endl; std::cout << "error: " << this->status.error_message() << std::endl;
} }
this->status = status; if (this->done) {
return;
}
this->terminateCallback();
try {
this->validate();
} catch (std::runtime_error &e) {
this->status = grpc::Status(grpc::StatusCode::INTERNAL, e.what());
}
this->done = true; this->done = true;
this->StartWritesDone(); this->StartWritesDone();
} }
template <class Request, class Response> template <class Request, class Response>
bool ClientWriteReactorBase<Request, Response>::isDone() { bool ClientWriteReactorBase<Request, Response>::isDone() {
return this->done; return this->done;
} }
Show All 11 Lines
View Options

services/base-image/Dockerfile

#include <grpcpp/grpcpp.h> #include <grpcpp/grpcpp.h>
namespace comm { namespace comm {
namespace network { namespace network {
namespace reactor { namespace reactor {
template <class Request, class Response> template <class Request, class Response>
class ClientWriteReactorBase : public grpc::ClientWriteReactor<Request> { class ClientWriteReactorBase : public grpc::ClientWriteReactor<Request> {
grpc::Status status; grpc::Status status = grpc::Status::OK;
bool done = false; bool done = false;
bool initialized = 0; bool initialized = 0;
Request request; Request request;
public: public:
Response response; Response response;
grpc::ClientContext context; grpc::ClientContext context;
void nextWrite(); void nextWrite();
void OnWriteDone(bool ok) override; void OnWriteDone(bool ok) override;
void terminate(const grpc::Status &status); void terminate(const grpc::Status &status);
bool isDone(); bool isDone();
void OnDone(const grpc::Status &status) override; void OnDone(const grpc::Status &status) override;
virtual std::unique_ptr<grpc::Status> prepareRequest(Request &request) = 0; virtual std::unique_ptr<grpc::Status> prepareRequest(Request &request) = 0;
virtual void validate(){};
virtual void doneCallback(){}; virtual void doneCallback(){};
virtual void terminateCallback(){}; virtual void terminateCallback(){};
}; };
template <class Request, class Response> template <class Request, class Response>
void ClientWriteReactorBase<Request, Response>::nextWrite() { void ClientWriteReactorBase<Request, Response>::nextWrite() {
this->request = Request(); this->request = Request();
std::unique_ptr<grpc::Status> status = this->prepareRequest(this->request); std::unique_ptr<grpc::Status> status = this->prepareRequest(this->request);
Show All 15 Lines if (!ok) {
return; return;
} }
this->nextWrite(); this->nextWrite();
} }
template <class Request, class Response> template <class Request, class Response>
void ClientWriteReactorBase<Request, Response>::terminate( void ClientWriteReactorBase<Request, Response>::terminate(
const grpc::Status &status) { const grpc::Status &status) {
this->terminateCallback(); if (this->status.ok()) {
if (this->done) { this->status = status;
return;
} }
if (!this->status.ok()) { if (!this->status.ok()) {
std::cout << "error: " << this->status.error_message() << std::endl; std::cout << "error: " << this->status.error_message() << std::endl;
} }
this->status = status; if (this->done) {
return;
}
this->terminateCallback();
try {
this->validate();
} catch (std::runtime_error &e) {
this->status = grpc::Status(grpc::StatusCode::INTERNAL, e.what());
}
this->done = true; this->done = true;
this->StartWritesDone(); this->StartWritesDone();
} }
template <class Request, class Response> template <class Request, class Response>
bool ClientWriteReactorBase<Request, Response>::isDone() { bool ClientWriteReactorBase<Request, Response>::isDone() {
return this->done; return this->done;
} }
Show All 11 Lines
View Options

services/base-image/contents/install_grpc.sh

Loading...
View Options

services/blob/docker-server/contents/build_server.sh

Loading...
View Options

services/blob/docker-server/contents/server/cmake-components/grpc.cmake

Loading...
View Options

services/tunnelbroker/CMakeLists.txt

Loading...
View Options

services/tunnelbroker/Dockerfile

Loading...