Paths

Table of Contentst

-
keyserver/src/uploads/
-
src/
-
uploads/
-
uploads.js
-
lib/
-
actions/
-
upload-actions.js
-
types/
-
media-types.js

[keyserver] Return blob hash and holder from upload endpoint
AbandonedPublic
Actions

Authored by bartek on Jun 20 2023, 5:20 AM.

Details

Reviewers

atul
ashoat
michal

Summary

Until now, the blob service upload endpoint (upload-media-metadata) returned the same result as the upload-multimedia endpoint.
However, it turns out to be convenient to have the blob hash and holder returned as well to avoid hacks (by passing them some other way) - it will be used in web client later (for blob deletion).

Depends on D8256

Test Plan

Called the upload-media-metadata endpoint from web input-state-container. Received the following output:

Diff Detail

Repository

rCOMM Comm

Lint

No Lint Coverage

Unit

No Test Coverage

Event Timeline

bartek created this revision.Jun 20 2023, 5:20 AM

bartek held this revision as a draft.

Herald added a subscriber: tomek. · View Herald TranscriptJun 20 2023, 5:20 AM

bartek edited the test plan for this revision. (Show Details)Jun 20 2023, 5:30 AM

bartek attached a referenced file: F595875: Screenshot 2023-06-20 at 13.44.14.png. (Show Details)

Harbormaster failed remote builds in B20343: Diff 27884!Jun 20 2023, 5:31 AM

Fix lint (broken rebase)

Harbormaster completed remote builds in B20344: Diff 27885.Jun 20 2023, 6:03 AM

Get rid of Flow workaround

bartek published this revision for review.Jun 20 2023, 6:23 AM

However, it turns out to be convenient to have the blob hash and holder returned as well to avoid hacks (by passing them some other way) - it will be used in web client later (for blob deletion).

Isn't this data already available on the client? If the data comes from the client, then I'm a little confused why we're returning it from the endpoint

Harbormaster completed remote builds in B20345: Diff 27886.Jun 20 2023, 6:26 AM

In D8257#243669, @ashoat wrote:

However, it turns out to be convenient to have the blob hash and holder returned as well to avoid hacks (by passing them some other way) - it will be used in web client later (for blob deletion).

Isn't this data already available on the client? If the data comes from the client, then I'm a little confused why we're returning it from the endpoint

Yes, it is. I don't know the beginnings of this convention, but we're doing the same with other values like loop, dimensions etc (probably the media could be processed on keyserver and these values could change).

Either way, I need to return UploadMediaMetadataResult from blobServiceUpload() in web/input-state-container. Alternatively, I can do it directly from there, just thought it would be cleaner to do it the same way other params are passed

Yes, it is. I don't know the beginnings of this convention, but we're doing the same with other values like loop, dimensions etc (probably the media could be processed on keyserver and these values could change).

I suspect that these other parameters either can be changed by the keyserver, or (perhaps more likely) at some point we thought we would want to be able to change them on the keyserver.

Either way, I need to return UploadMediaMetadataResult from blobServiceUpload() in web/input-state-container. Alternatively, I can do it directly from there, just thought it would be cleaner to do it the same way other params are passed

If we are confident that the keyserver will not change this parameter, then I think we should avoid wasting network traffic on it. My impression is that the keyserver won't be downloading the whole blob anyways, so there's no way to confirm the blob hash (or generate a new one).

In D8257#243677, @ashoat wrote:

If we are confident that the keyserver will not change this parameter, then I think we should avoid wasting network traffic on it. My impression is that the keyserver won't be downloading the whole blob anyways, so there's no way to confirm the blob hash (or generate a new one).

Blob hash won't be changed on keyserver, and regarding the holder: it's not gonna be changed too, unless we'd like to introduce some holder-manipulation logic in the future (rather unlikely in this case) so I'll do the other approach

bartek mentioned this in D8320: [web] Use local holder for blob deletion.Jun 26 2023, 11:08 PM

Decided to go with a client-side approach in D8320 so abandoning this for now. We can always return additional data in the future without breaking if needed.

bartek mentioned this in rCOMM9310a1ad5108: [web] Use local holder for blob deletion.Jun 28 2023, 11:12 PM

Revision Contents
Changeset List

Path

Size

keyserver/

src/

uploads/

uploads.js

11 lines

lib/

actions/

upload-actions.js

7 lines

types/

media-types.js

12 lines

Diff 27884

View Options

keyserver/src/uploads/uploads.js

Show First 20 Lines • Show All 342 Lines • ▼ Show 20 Lines	query.append(SQL`
ORDER BY tw.threadID, tw.time DESC, tw.id DESC		ORDER BY tw.threadID, tw.time DESC, tw.id DESC
`);		`);

const [result] = await dbQuery(query);		const [result] = await dbQuery(query);
const derivedMessages = await fetchDerivedMessages(result, viewer);		const derivedMessages = await fetchDerivedMessages(result, viewer);
const messages = await parseMessageSQLResult(result, derivedMessages, viewer);		const messages = await parseMessageSQLResult(result, derivedMessages, viewer);

const rawMessageInfos = [];		const rawMessageInfos = [];
const threadToMessageCount = new Map();		const threadToMessageCount = new Map<string, number>();
for (const message of messages) {		for (const message of messages) {
const { rawMessageInfo } = message;		const { rawMessageInfo } = message;
rawMessageInfos.push(rawMessageInfo);		rawMessageInfos.push(rawMessageInfo);
const { threadID } = rawMessageInfo;		const { threadID } = rawMessageInfo;
const currentCountValue = threadToMessageCount.get(threadID);		const currentCountValue = threadToMessageCount.get(threadID);
const currentCount = currentCountValue ? currentCountValue : 0;		const currentCount = currentCountValue ? currentCountValue : 0;
threadToMessageCount.set(threadID, currentCount + 1);		threadToMessageCount.set(threadID, currentCount + 1);
}		}
▲ Show 20 Lines • Show All 479 Lines • ▼ Show 20 Lines	SELECT m.id, (
ORDER BY time DESC, id DESC		ORDER BY time DESC, id DESC
LIMIT 1		LIMIT 1
) content		) content
FROM messages m		FROM messages m
WHERE m.id IN(${messageIDs})		WHERE m.id IN(${messageIDs})
`;		`;

const [result] = await dbQuery(latestEditedMessageQuery);		const [result] = await dbQuery(latestEditedMessageQuery);
const latestContentByID = new Map();		const latestContentByID = new Map<string, EditMessageContent>();
for (const row of result) {		for (const row of result) {
if (!row.content) {		if (!row.content) {
continue;		continue;
}		}
const content = JSON.parse(row.content);		const content = JSON.parse(row.content);
latestContentByID.set(row.id.toString(), content);		latestContentByID.set(row.id.toString(), content);
}		}

▲ Show 20 Lines • Show All 181 Lines • Show Last 20 Lines

View Options

lib/actions/upload-actions.js

Show First 20 Lines • Show All 342 Lines • ▼ Show 20 Lines	query.append(SQL`
ORDER BY tw.threadID, tw.time DESC, tw.id DESC		ORDER BY tw.threadID, tw.time DESC, tw.id DESC
`);		`);

const [result] = await dbQuery(query);		const [result] = await dbQuery(query);
const derivedMessages = await fetchDerivedMessages(result, viewer);		const derivedMessages = await fetchDerivedMessages(result, viewer);
const messages = await parseMessageSQLResult(result, derivedMessages, viewer);		const messages = await parseMessageSQLResult(result, derivedMessages, viewer);

const rawMessageInfos = [];		const rawMessageInfos = [];
const threadToMessageCount = new Map();		const threadToMessageCount = new Map<string, number>();
for (const message of messages) {		for (const message of messages) {
const { rawMessageInfo } = message;		const { rawMessageInfo } = message;
rawMessageInfos.push(rawMessageInfo);		rawMessageInfos.push(rawMessageInfo);
const { threadID } = rawMessageInfo;		const { threadID } = rawMessageInfo;
const currentCountValue = threadToMessageCount.get(threadID);		const currentCountValue = threadToMessageCount.get(threadID);
const currentCount = currentCountValue ? currentCountValue : 0;		const currentCount = currentCountValue ? currentCountValue : 0;
threadToMessageCount.set(threadID, currentCount + 1);		threadToMessageCount.set(threadID, currentCount + 1);
}		}
▲ Show 20 Lines • Show All 479 Lines • ▼ Show 20 Lines	SELECT m.id, (
ORDER BY time DESC, id DESC		ORDER BY time DESC, id DESC
LIMIT 1		LIMIT 1
) content		) content
FROM messages m		FROM messages m
WHERE m.id IN(${messageIDs})		WHERE m.id IN(${messageIDs})
`;		`;

const [result] = await dbQuery(latestEditedMessageQuery);		const [result] = await dbQuery(latestEditedMessageQuery);
const latestContentByID = new Map();		const latestContentByID = new Map<string, EditMessageContent>();
for (const row of result) {		for (const row of result) {
if (!row.content) {		if (!row.content) {
continue;		continue;
}		}
const content = JSON.parse(row.content);		const content = JSON.parse(row.content);
latestContentByID.set(row.id.toString(), content);		latestContentByID.set(row.id.toString(), content);
}		}

▲ Show 20 Lines • Show All 181 Lines • Show Last 20 Lines

View Options

lib/types/media-types.js

Show First 20 Lines • Show All 342 Lines • ▼ Show 20 Lines	query.append(SQL`
ORDER BY tw.threadID, tw.time DESC, tw.id DESC		ORDER BY tw.threadID, tw.time DESC, tw.id DESC
`);		`);

const [result] = await dbQuery(query);		const [result] = await dbQuery(query);
const derivedMessages = await fetchDerivedMessages(result, viewer);		const derivedMessages = await fetchDerivedMessages(result, viewer);
const messages = await parseMessageSQLResult(result, derivedMessages, viewer);		const messages = await parseMessageSQLResult(result, derivedMessages, viewer);

const rawMessageInfos = [];		const rawMessageInfos = [];
const threadToMessageCount = new Map();		const threadToMessageCount = new Map<string, number>();
for (const message of messages) {		for (const message of messages) {
const { rawMessageInfo } = message;		const { rawMessageInfo } = message;
rawMessageInfos.push(rawMessageInfo);		rawMessageInfos.push(rawMessageInfo);
const { threadID } = rawMessageInfo;		const { threadID } = rawMessageInfo;
const currentCountValue = threadToMessageCount.get(threadID);		const currentCountValue = threadToMessageCount.get(threadID);
const currentCount = currentCountValue ? currentCountValue : 0;		const currentCount = currentCountValue ? currentCountValue : 0;
threadToMessageCount.set(threadID, currentCount + 1);		threadToMessageCount.set(threadID, currentCount + 1);
}		}
▲ Show 20 Lines • Show All 479 Lines • ▼ Show 20 Lines	SELECT m.id, (
ORDER BY time DESC, id DESC		ORDER BY time DESC, id DESC
LIMIT 1		LIMIT 1
) content		) content
FROM messages m		FROM messages m
WHERE m.id IN(${messageIDs})		WHERE m.id IN(${messageIDs})
`;		`;

const [result] = await dbQuery(latestEditedMessageQuery);		const [result] = await dbQuery(latestEditedMessageQuery);
const latestContentByID = new Map();		const latestContentByID = new Map<string, EditMessageContent>();
for (const row of result) {		for (const row of result) {
if (!row.content) {		if (!row.content) {
continue;		continue;
}		}
const content = JSON.parse(row.content);		const content = JSON.parse(row.content);
latestContentByID.set(row.id.toString(), content);		latestContentByID.set(row.id.toString(), content);
}		}

▲ Show 20 Lines • Show All 181 Lines • Show Last 20 Lines

[keyserver] Return blob hash and holder from upload endpointAbandonedPublicActions

Details

Diff Detail

Event Timeline

Revision ContentsChangeset List

Diff 27884

keyserver/src/uploads/uploads.js

lib/actions/upload-actions.js

lib/types/media-types.js

[keyserver] Return blob hash and holder from upload endpoint
AbandonedPublic
Actions

Revision Contents
Changeset List