Page MenuHomePhabricator
Differential D3159

[services] Backup - Add Tools
ClosedPublic
Actions

Authored by • karol on Feb 10 2022, 6:28 AM.
Tags
None
Referenced Files
Unknown Object (File)
Fri, Mar 7, 4:22 PM
Unknown Object (File)
Fri, Mar 7, 4:20 PM
Unknown Object (File)
Tue, Feb 25, 3:51 PM
Unknown Object (File)
Sat, Feb 22, 8:36 AM
Unknown Object (File)
Feb 3 2025, 5:36 AM
Unknown Object (File)
Jan 27 2025, 2:47 PM
Unknown Object (File)
Jan 24 2025, 7:53 PM
Unknown Object (File)
Jan 17 2025, 9:40 AM
View All Files
Subscribers
• adrian
ashoat
atul
• benschac
tomek

Details

Reviewers
tomek
• max
varun
ashoat
• jim
Commits
rCOMM073083fa62b2: [services] Backup - Add Tools
Summary

Adding Tools with getCurrentTimestamp which will be needed for creating backup items.

Depends on D3158

Test Plan

Backup service builds

Diff Detail

Repository
rCOMM Comm
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

• karol created this revision.Feb 10 2022, 6:28 AM
Herald added subscribers: atul, • adrian, tomek, ashoat. · View Herald TranscriptFeb 10 2022, 6:28 AM
Harbormaster completed remote builds in B6709: Diff 9519.Feb 10 2022, 6:35 AM
• karol requested review of this revision.Feb 10 2022, 6:35 AM
• karol edited the summary of this revision. (Show Details)Feb 10 2022, 6:35 AM
• karol edited the test plan for this revision. (Show Details)
• karol added reviewers: tomek, • max, varun.
• karol added a parent revision: D3158: [services] Backup - Implementation structure.
• karol added a child revision: D3160: [services] Backup - Add DB operations for Backup.Feb 10 2022, 6:38 AM
tomek accepted this revision.Feb 11 2022, 2:01 AM
This revision is now accepted and ready to land.Feb 11 2022, 2:01 AM
Herald added a reviewer: ashoat. · View Herald TranscriptFeb 11 2022, 2:01 AM
Herald added a subscriber: • benschac. · View Herald Transcript
This revision now requires review to proceed.Feb 11 2022, 2:01 AM
ashoat accepted this revision.Feb 13 2022, 8:02 PM
ashoat added a reviewer: • jim.

We have the exact same function in services/blob/docker-server/contents/server/src/Tools.cpp and services/tunnelbroker/docker-server/contents/server/src/Tools/Tools.cpp. I assume we're tracking fixing this in ENG-324... @karol-bisztyga, before landing can you add a comment there to make sure we remember to dedup all of the copy-pasted getCurrentTimestamp functions?

This revision is now accepted and ready to land.Feb 13 2022, 8:02 PM
• karol added a comment.Feb 14 2022, 1:36 AM

This task will be resolved after I finish the backup service. I'm going to look at the services and try to exclude as many mutual items as possible to the "lib" space. I remember about this and keep it in mind.

Closed by commit rCOMM073083fa62b2: [services] Backup - Add Tools (authored by • karol). · Explain WhyMar 4 2022, 4:01 AM
This revision was automatically updated to reflect the committed changes.
• karol added a commit: rCOMM073083fa62b2: [services] Backup - Add Tools.

Revision Contents
Changeset List

PathSize
services/
backup/
docker-server/
contents/
server/
src/
11 lines
15 lines

Diff 10060

View Options

services/backup/docker-server/contents/server/src/Tools.h

  • This file was copied to services/backup/src/Reactors/client/blob/BlobPutClientReactor.cpp.
#pragma once #pragma once
#include "ClientBidiReactorBase.h"
#include "Constants.h" #include "Constants.h"
#include "../_generated/blob.grpc.pb.h" #include "../_generated/blob.grpc.pb.h"
#include "../_generated/blob.pb.h" #include "../_generated/blob.pb.h"
#include "ClientBidiReactorBase.h"
#include <folly/MPMCQueue.h> #include <folly/MPMCQueue.h>
#include <grpcpp/grpcpp.h> #include <grpcpp/grpcpp.h>
#include <condition_variable> #include <condition_variable>
#include <iostream>
#include <memory> #include <memory>
#include <string> #include <string>
namespace comm { namespace comm {
namespace network { namespace network {
namespace reactor { namespace reactor {
class BlobPutClientReactor class BlobPutClientReactor
Show All 22 Linespublic:
void scheduleSendingDataChunk(std::unique_ptr<std::string> dataChunk); void scheduleSendingDataChunk(std::unique_ptr<std::string> dataChunk);
std::unique_ptr<grpc::Status> prepareRequest( std::unique_ptr<grpc::Status> prepareRequest(
blob::PutRequest &request, blob::PutRequest &request,
std::shared_ptr<blob::PutResponse> previousResponse) override; std::shared_ptr<blob::PutResponse> previousResponse) override;
void doneCallback() override; void doneCallback() override;
grpc::Status getStatus() const; grpc::Status getStatus() const;
}; };
BlobPutClientReactor::BlobPutClientReactor(
const std::string &holder,
const std::string &hash,
std::condition_variable *terminationNotifier)
: holder(holder),
hash(hash),
dataChunks(folly::MPMCQueue<std::string>(100)),
terminationNotifier(terminationNotifier) {
}
void BlobPutClientReactor::scheduleSendingDataChunk(
std::unique_ptr<std::string> dataChunk) {
if (!this->dataChunks.write(std::move(*dataChunk))) {
throw std::runtime_error(
"Error scheduling sending a data chunk to send to the blob service");
}
}
std::unique_ptr<grpc::Status> BlobPutClientReactor::prepareRequest(
blob::PutRequest &request,
std::shared_ptr<blob::PutResponse> previousResponse) {
if (this->state == State::SEND_HOLDER) {
this->request.set_holder(this->holder);
this->state = State::SEND_HASH;
return nullptr;
}
if (this->state == State::SEND_HASH) {
request.set_blobhash(this->hash);
this->state = State::SEND_CHUNKS;
return nullptr;
}
if (previousResponse->dataexists()) {
return std::make_unique<grpc::Status>(grpc::Status::OK);
}
std::string dataChunk;
this->dataChunks.blockingRead(dataChunk);
if (dataChunk.empty()) {
return std::make_unique<grpc::Status>(grpc::Status::OK);
}
request.set_datachunk(dataChunk);
return nullptr;
}
void BlobPutClientReactor::doneCallback() {
this->terminationNotifier->notify_one();
}
grpc::Status BlobPutClientReactor::getStatus() const {
return this->status;
}
} // namespace reactor } // namespace reactor
} // namespace network } // namespace network
} // namespace comm } // namespace comm
View Options

services/backup/docker-server/contents/server/src/Tools.cpp

  • This file was copied to services/backup/src/Reactors/client/blob/BlobPutClientReactor.cpp.
#pragma once #pragma once
#include "ClientBidiReactorBase.h"
#include "Constants.h" #include "Constants.h"
#include "../_generated/blob.grpc.pb.h" #include "../_generated/blob.grpc.pb.h"
#include "../_generated/blob.pb.h" #include "../_generated/blob.pb.h"
#include "ClientBidiReactorBase.h"
#include <folly/MPMCQueue.h> #include <folly/MPMCQueue.h>
#include <grpcpp/grpcpp.h> #include <grpcpp/grpcpp.h>
#include <condition_variable> #include <condition_variable>
#include <iostream>
#include <memory> #include <memory>
#include <string> #include <string>
namespace comm { namespace comm {
namespace network { namespace network {
namespace reactor { namespace reactor {
class BlobPutClientReactor class BlobPutClientReactor
Show All 22 Linespublic:
void scheduleSendingDataChunk(std::unique_ptr<std::string> dataChunk); void scheduleSendingDataChunk(std::unique_ptr<std::string> dataChunk);
std::unique_ptr<grpc::Status> prepareRequest( std::unique_ptr<grpc::Status> prepareRequest(
blob::PutRequest &request, blob::PutRequest &request,
std::shared_ptr<blob::PutResponse> previousResponse) override; std::shared_ptr<blob::PutResponse> previousResponse) override;
void doneCallback() override; void doneCallback() override;
grpc::Status getStatus() const; grpc::Status getStatus() const;
}; };
BlobPutClientReactor::BlobPutClientReactor(
const std::string &holder,
const std::string &hash,
std::condition_variable *terminationNotifier)
: holder(holder),
hash(hash),
dataChunks(folly::MPMCQueue<std::string>(100)),
terminationNotifier(terminationNotifier) {
}
void BlobPutClientReactor::scheduleSendingDataChunk(
std::unique_ptr<std::string> dataChunk) {
if (!this->dataChunks.write(std::move(*dataChunk))) {
throw std::runtime_error(
"Error scheduling sending a data chunk to send to the blob service");
}
}
std::unique_ptr<grpc::Status> BlobPutClientReactor::prepareRequest(
blob::PutRequest &request,
std::shared_ptr<blob::PutResponse> previousResponse) {
if (this->state == State::SEND_HOLDER) {
this->request.set_holder(this->holder);
this->state = State::SEND_HASH;
return nullptr;
}
if (this->state == State::SEND_HASH) {
request.set_blobhash(this->hash);
this->state = State::SEND_CHUNKS;
return nullptr;
}
if (previousResponse->dataexists()) {
return std::make_unique<grpc::Status>(grpc::Status::OK);
}
std::string dataChunk;
this->dataChunks.blockingRead(dataChunk);
if (dataChunk.empty()) {
return std::make_unique<grpc::Status>(grpc::Status::OK);
}
request.set_datachunk(dataChunk);
return nullptr;
}
void BlobPutClientReactor::doneCallback() {
this->terminationNotifier->notify_one();
}
grpc::Status BlobPutClientReactor::getStatus() const {
return this->status;
}
} // namespace reactor } // namespace reactor
} // namespace network } // namespace network
} // namespace comm } // namespace comm