Page MenuHomePhabricator
Differential D3594

[services] Blob - Fix reading data from S3
ClosedPublic
Actions

Authored by • karol on Apr 1 2022, 3:07 AM.
Tags
None
Referenced Files
F5112515: D3594.diff
Wed, Apr 2, 5:08 AM
Unknown Object (File)
Mon, Mar 31, 5:10 PM
Unknown Object (File)
Mon, Mar 31, 4:09 PM
Unknown Object (File)
Sun, Mar 30, 9:50 PM
Unknown Object (File)
Sun, Mar 30, 8:15 PM
Unknown Object (File)
Sat, Mar 29, 9:26 PM
Unknown Object (File)
Mon, Mar 17, 4:06 PM
Unknown Object (File)
Mon, Mar 17, 6:16 AM
View All Files
Subscribers
• abosh
• adrian
ashoat
atul
• benschac
tomek

Details

Reviewers
tomek
varun
• max
• jim
Commits
rCOMM656be3bc4746: [services] Blob - Fix reading data from S3
Summary

Depends on D3651

Raised in https://phabricator.ashoat.com/D3470#97072

We want to fix reading the S3 object so it doesn't cut the stream on \n

Test Plan
cd services
yarn test-blob-service

Diff Detail

Repository
rCOMM Comm
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

• karol created this revision.Apr 1 2022, 3:07 AM
Herald added subscribers: • abosh, • benschac, atul and 3 others. · View Herald TranscriptApr 1 2022, 3:07 AM
• karol mentioned this in D3470: [services] Blob - Add server reactor implementations - get reactor.Apr 1 2022, 3:11 AM
Harbormaster completed remote builds in B7791: Diff 10933.Apr 1 2022, 3:14 AM
• karol requested review of this revision.Apr 1 2022, 3:14 AM
• karol edited the summary of this revision. (Show Details)Apr 1 2022, 3:17 AM
• karol edited the test plan for this revision. (Show Details)
• karol added reviewers: tomek, varun, • max, • jim.
• karol added a parent revision: D3587: [services] Backup - Rename mutex for create new backup.
• karol added a child revision: D3596: [services] Blob - Fix test name for storage manager tests.
tomek accepted this revision.Apr 1 2022, 3:38 AM
tomek added inline comments.
services/blob/src/AwsS3Bucket.cpp
122–123 ↗(On Diff #10933)

Why do we need to resize?

This revision is now accepted and ready to land.Apr 1 2022, 3:38 AM
• karol edited the summary of this revision. (Show Details)Apr 7 2022, 1:20 AM
• karol added a parent revision: D3586: [services] Backup - Send log - add store in database.
• karol edited the summary of this revision. (Show Details)Apr 7 2022, 7:23 AM
• karol added a parent revision: D3651: [services] Backup - Send log - finish read request logic.
• karol added inline comments.Apr 13 2022, 4:53 AM
services/blob/src/AwsS3Bucket.cpp
122–123 ↗(On Diff #10933)

We don't. It is a leftover. It was used to allocate the mem up front.

• karol updated this revision to Diff 11397.Apr 13 2022, 4:55 AM

remove resize

This revision was landed with ongoing or failed builds.Apr 13 2022, 4:58 AM
Closed by commit rCOMM656be3bc4746: [services] Blob - Fix reading data from S3 (authored by • karol). · Explain Why
This revision was automatically updated to reflect the committed changes.
• karol added a commit: rCOMM656be3bc4746: [services] Blob - Fix reading data from S3.
Harbormaster completed remote builds in B8151: Diff 11397.Apr 13 2022, 4:59 AM

Revision Contents
Changeset List

PathSize
services/
blob/
src/
13 lines
Reactors/
server/
7 lines

Diff 11400

View Options

services/blob/src/AwsS3Bucket.cpp

Show First 20 Lines • Show All 98 Lines • ▼ Show 20 Lines
export type FFmpegStatistics = { export type FFmpegStatistics = {
// seconds of video being processed per second // seconds of video being processed per second
+speed: number, +speed: number,
// total milliseconds of video processed so far // total milliseconds of video processed so far
+time: number, +time: number,
// total result file size in bytes so far // total result file size in bytes so far
+size: number, +size: number,
+videoQuality: number,
+videoFrameNumber: number,
+videoFps: number,
+bitrate: number,
}; };
export type TranscodeVideoMediaMissionStep = { export type TranscodeVideoMediaMissionStep = {
+step: 'video_ffmpeg_transcode', +step: 'video_ffmpeg_transcode',
+success: boolean, +success: boolean,
+exceptionMessage: ?string, +exceptionMessage: ?string,
+time: number, // ms +time: number, // ms
+returnCode: ?number,
+newPath: ?string, +newPath: ?string,
+stats: ?FFmpegStatistics, +stats: ?FFmpegStatistics,
}; };
export type VideoGenerateThumbnailMediaMissionStep = { export type VideoGenerateThumbnailMediaMissionStep = {
+step: 'video_generate_thumbnail', +step: 'video_generate_thumbnail',
+success: boolean, +success: boolean,
+time: number, // ms +time: number, // ms
▲ Show 20 Lines • Show All 698 Lines • Show Last 20 Lines
View Options

services/blob/src/Reactors/server/GetReactor.h

Show First 20 Lines • Show All 98 Lines • ▼ Show 20 Lines
export type FFmpegStatistics = { export type FFmpegStatistics = {
// seconds of video being processed per second // seconds of video being processed per second
+speed: number, +speed: number,
// total milliseconds of video processed so far // total milliseconds of video processed so far
+time: number, +time: number,
// total result file size in bytes so far // total result file size in bytes so far
+size: number, +size: number,
+videoQuality: number,
+videoFrameNumber: number,
+videoFps: number,
+bitrate: number,
}; };
export type TranscodeVideoMediaMissionStep = { export type TranscodeVideoMediaMissionStep = {
+step: 'video_ffmpeg_transcode', +step: 'video_ffmpeg_transcode',
+success: boolean, +success: boolean,
+exceptionMessage: ?string, +exceptionMessage: ?string,
+time: number, // ms +time: number, // ms
+returnCode: ?number,
+newPath: ?string, +newPath: ?string,
+stats: ?FFmpegStatistics, +stats: ?FFmpegStatistics,
}; };
export type VideoGenerateThumbnailMediaMissionStep = { export type VideoGenerateThumbnailMediaMissionStep = {
+step: 'video_generate_thumbnail', +step: 'video_generate_thumbnail',
+success: boolean, +success: boolean,
+time: number, // ms +time: number, // ms
▲ Show 20 Lines • Show All 698 Lines • Show Last 20 Lines