Page MenuHomePhabricator
Differential D5018

[services] Backup - Use Blob client in c++ - Implement add attachment
ClosedPublic
Actions

Authored by • karol on Sep 1 2022, 4:10 AM.
Tags
None
Referenced Files
Unknown Object (File)
Tue, Apr 8, 11:25 PM
Unknown Object (File)
Wed, Apr 2, 6:18 AM
Unknown Object (File)
Wed, Apr 2, 6:18 AM
Unknown Object (File)
Tue, Apr 1, 8:33 PM
Unknown Object (File)
Tue, Apr 1, 4:35 PM
Unknown Object (File)
Tue, Apr 1, 4:35 PM
Unknown Object (File)
Tue, Apr 1, 4:35 PM
Unknown Object (File)
Tue, Apr 1, 4:35 PM
View All Files
Subscribers
• abosh
ashoat
atul
tomek

Details

Reviewers
• jon
tomek
• max
Commits
rCOMM0b822963c7ec: [services] Backup - Use Blob client in c++ - Implement add attachment
Summary

Depends on D5017

Adding c++ implementation for using the blob client in the backup reactor for adding attachments.

Test Plan

backup builds

Diff Detail

Repository
rCOMM Comm
Lint
No Lint Coverage
Unit
No Test Coverage

Event Timeline

• karol created this revision.Sep 1 2022, 4:10 AM
Herald added a reviewer: • jon. · View Herald TranscriptSep 1 2022, 4:10 AM
Herald added subscribers: • abosh, atul, tomek, ashoat. · View Herald Transcript
• karol edited the summary of this revision. (Show Details)Sep 1 2022, 4:26 AM
• karol edited the test plan for this revision. (Show Details)
• karol added reviewers: tomek, • max.
• karol added a parent revision: D5017: [services] Backup - Use Blob client in c++ - Implement create new backup.
• karol added a child revision: D5019: [services] Backup - Use Blob client in c++ - Implement send log.
Harbormaster completed remote builds in B11775: Diff 16178.Sep 1 2022, 4:29 AM
• karol requested review of this revision.Sep 1 2022, 4:29 AM
• karol planned changes to this revision.Sep 2 2022, 5:47 AM
• karol updated this revision to Diff 16246.Sep 2 2022, 5:48 AM

rebase

Harbormaster failed remote builds in B11841: Diff 16246!Sep 2 2022, 5:52 AM
tomek requested changes to this revision.Sep 5 2022, 5:56 AM
tomek added inline comments.
services/backup/src/Reactors/server/AddAttachmentsUtility.cpp
78–80 ↗(On Diff #16246)

Doesn't seem like parentheses pair here.

backup builds

Are you sure?

This revision now requires changes to proceed.Sep 5 2022, 5:56 AM
• karol updated this revision to Diff 16355.Sep 6 2022, 3:22 AM

fix

services/backup/src/Reactors/server/AddAttachmentsUtility.cpp
78–80 ↗(On Diff #16246)

ok thx not sure how this happened, will fix

Harbormaster completed remote builds in B11912: Diff 16355.Sep 6 2022, 3:32 AM
• karol planned changes to this revision.Sep 7 2022, 2:32 AM

remove the comments

• karol updated this revision to Diff 16396.Sep 7 2022, 2:34 AM

remove comments

tomek accepted this revision.Sep 7 2022, 5:24 AM
tomek added inline comments.
services/backup/src/Reactors/server/AddAttachmentsUtility.cpp
81 ↗(On Diff #16396)

Is there a downside of calling one write after another without the read between them?

This revision is now accepted and ready to land.Sep 7 2022, 5:24 AM
• karol added inline comments.Sep 8 2022, 3:48 AM
services/backup/src/Reactors/server/AddAttachmentsUtility.cpp
81 ↗(On Diff #16396)

I think I explained this in another task. Anyway, the thing is, this is how grpc works, it yields a response even if it is empty. Our response queue is limited to 1 (after that, it will hang), so we should always read using the bidi stream, no matter if we use the response or not. We can, of course, add additional logic that would automatically discard responses.

• karol updated this revision to Diff 16450.Sep 8 2022, 4:01 AM

rebase

Harbormaster failed remote builds in B11995: Diff 16450!Sep 8 2022, 4:09 AM
This revision was landed with ongoing or failed builds.Sep 8 2022, 4:10 AM
Closed by commit rCOMM0b822963c7ec: [services] Backup - Use Blob client in c++ - Implement add attachment (authored by • karol). · Explain Why
This revision was automatically updated to reflect the committed changes.
• karol added a commit: rCOMM0b822963c7ec: [services] Backup - Use Blob client in c++ - Implement add attachment.

Revision Contents
Changeset List

PathSize
services/
backup/
src/
Reactors/
server/
25 lines

Diff 16450

View Options

services/backup/src/Reactors/server/AddAttachmentsUtility.cpp

#include "AddAttachmentsUtility.h" #include "AddAttachmentsUtility.h"
#include "blob_client/src/lib.rs.h"
#include <glog/logging.h> #include <glog/logging.h>
#include "BackupItem.h" #include "BackupItem.h"
#include "Constants.h" #include "Constants.h"
#include "DatabaseManager.h" #include "DatabaseManager.h"
#include "Tools.h" #include "Tools.h"
namespace comm { namespace comm {
Show All 33 Lines if (logID.empty()) {
if (!logItem->getPersistedInBlob() && if (!logItem->getPersistedInBlob() &&
database::LogItem::getItemSize(logItem.get()) > database::LogItem::getItemSize(logItem.get()) >
LOG_DATA_SIZE_DATABASE_LIMIT) { LOG_DATA_SIZE_DATABASE_LIMIT) {
bool old = logItem->getPersistedInBlob(); bool old = logItem->getPersistedInBlob();
logItem = this->moveToS3(logItem); logItem = this->moveToS3(logItem);
} }
database::DatabaseManager::getInstance().putLogItem(*logItem); database::DatabaseManager::getInstance().putLogItem(*logItem);
} }
} catch (std::runtime_error &e) { } catch (std::exception &e) {
LOG(ERROR) << e.what(); LOG(ERROR) << e.what();
status = grpc::Status(grpc::StatusCode::INTERNAL, e.what()); status = grpc::Status(grpc::StatusCode::INTERNAL, e.what());
} }
return status; return status;
} }
std::shared_ptr<database::LogItem> std::shared_ptr<database::LogItem>
AddAttachmentsUtility::moveToS3(std::shared_ptr<database::LogItem> logItem) { AddAttachmentsUtility::moveToS3(std::shared_ptr<database::LogItem> logItem) {
std::string holder = tools::generateHolder( std::string holder = tools::generateHolder(
logItem->getDataHash(), logItem->getBackupID(), logItem->getLogID()); logItem->getDataHash(), logItem->getBackupID(), logItem->getLogID());
std::string data = std::move(logItem->getValue()); std::string data = std::move(logItem->getValue());
std::shared_ptr<database::LogItem> newLogItem = std::shared_ptr<database::LogItem> newLogItem =
std::make_shared<database::LogItem>( std::make_shared<database::LogItem>(
logItem->getBackupID(), logItem->getBackupID(),
logItem->getLogID(), logItem->getLogID(),
true, true,
holder, holder,
logItem->getAttachmentHolders(), logItem->getAttachmentHolders(),
logItem->getDataHash()); logItem->getDataHash());
// put into S3 // put into S3
std::condition_variable blobPutDoneCV; std::condition_variable blobPutDoneCV;
std::mutex blobPutDoneCVMutex; std::mutex blobPutDoneCVMutex;
// todo:blob perform put put_client_initialize_cxx();
// todo:blob perform put:add chunk (std::move(data)) put_client_write_cxx(
// todo:blob perform put:add chunk ("") tools::getBlobPutField(blob::PutRequest::DataCase::kHolder),
// todo:blob perform put:wait for completion holder.c_str());
put_client_blocking_read_cxx();
put_client_write_cxx(
tools::getBlobPutField(blob::PutRequest::DataCase::kBlobHash),
newLogItem->getDataHash().c_str());
rust::String responseStr = put_client_blocking_read_cxx();
// data exists?
if (!(bool)tools::charPtrToInt(responseStr.c_str())) {
put_client_write_cxx(
tools::getBlobPutField(blob::PutRequest::DataCase::kDataChunk),
std::move(data).c_str());
put_client_blocking_read_cxx();
}
put_client_terminate_cxx();
return newLogItem; return newLogItem;
} }
} // namespace reactor } // namespace reactor
} // namespace network } // namespace network
} // namespace comm } // namespace comm