Page MenuHomePhabricator
Differential D8777

[services-lib] Introduce Blob Client
ClosedPublic
Actions

Authored by bartek on Aug 10 2023, 10:16 AM.
Tags
None
Referenced Files
Unknown Object (File)
Wed, Apr 2, 10:29 PM
Unknown Object (File)
Mon, Mar 31, 10:54 PM
Unknown Object (File)
Tue, Mar 25, 4:46 PM
Unknown Object (File)
Sun, Mar 23, 8:57 AM
Unknown Object (File)
Sat, Mar 22, 3:41 PM
Unknown Object (File)
Tue, Mar 18, 5:40 PM
Unknown Object (File)
Mon, Mar 17, 12:17 AM
Unknown Object (File)
Mar 6 2025, 1:42 AM
View All Files
Subscribers
ashoat
tomek

Details

Reviewers
michal
• jon
Commits
rCOMM9813647e8139: [services-lib] Introduce Blob Client
Summary

Part of ENG-4597.

This diff introduces a foundation for blob client in comm-services-lib. The client is hidden behind a crate feature so services that do not need it do not have to depend on it (e.g. feature-flags).

Test Plan

Cargo build in comm-services-lib, blob and feature-flags.

Diff Detail

Repository
rCOMM Comm
Lint
No Lint Coverage
Unit
No Test Coverage

Event Timeline

bartek created this revision.Aug 10 2023, 10:16 AM
bartek held this revision as a draft.
Herald added a reviewer: • jon. · View Herald TranscriptAug 10 2023, 10:16 AM
Herald added subscribers: tomek, ashoat. · View Herald Transcript
bartek added a child revision: D8778: [services-lib] Add method to download blobs.Aug 10 2023, 10:18 AM
Harbormaster completed remote builds in B21693: Diff 29817.Aug 10 2023, 10:34 AM
bartek updated this revision to Diff 29822.Aug 10 2023, 10:38 AM

Fix Cargo.lock dependencies in services

Harbormaster completed remote builds in B21698: Diff 29822.Aug 10 2023, 10:57 AM
bartek added a child revision: D8786: [services-lib] Add HTTP client auth support.Aug 10 2023, 3:29 PM
bartek removed a child revision: D8786: [services-lib] Add HTTP client auth support.Aug 10 2023, 3:34 PM
bartek added a child revision: D8791: [services-lib] Add BlobInfo utility type.Aug 10 2023, 3:47 PM
bartek added inline comments.Aug 10 2023, 3:48 PM
services/backup/Cargo.lock
592

These files changed because I added the dependency to the lib

services/comm-services-lib/src/blob/client.rs
1

Will be removed in the next diff

47–50

This is the place where the authorization token will be stored and managed.

services/comm-services-lib/src/blob/mod.rs
1–2

This is the first submodule, but there will be more added later

bartek published this revision for review.Aug 11 2023, 2:19 AM
michal accepted this revision.Aug 11 2023, 4:11 AM

This will be very useful! I would prefer to have multiple packages in comm-services-lib/ (comm-services-lib/database-utils/, comm-services-lib/blob-client), but I don't really have strong argument against using features for this.

This revision is now accepted and ready to land.Aug 11 2023, 4:11 AM
Closed by commit rCOMM9813647e8139: [services-lib] Introduce Blob Client (authored by bartek). · Explain WhyAug 16 2023, 7:25 AM
This revision was automatically updated to reflect the committed changes.
bartek added a commit: rCOMM9813647e8139: [services-lib] Introduce Blob Client.

Revision Contents
Changeset List

PathSize
services/
backup/
blob/
comm-services-lib/
20 lines
src/
blob/
60 lines
2 lines
1 line
feature-flags/

Diff 29822

View Options

services/backup/Cargo.lock

Show First 20 Lines • Show All 342 Lines • ▼ Show 20 Lines query.append(SQL`
ORDER BY tw.threadID, tw.time DESC, tw.id DESC ORDER BY tw.threadID, tw.time DESC, tw.id DESC
`); `);
const [result] = await dbQuery(query); const [result] = await dbQuery(query);
const derivedMessages = await fetchDerivedMessages(result, viewer); const derivedMessages = await fetchDerivedMessages(result, viewer);
const messages = await parseMessageSQLResult(result, derivedMessages, viewer); const messages = await parseMessageSQLResult(result, derivedMessages, viewer);
const rawMessageInfos = []; const rawMessageInfos = [];
const threadToMessageCount = new Map(); const threadToMessageCount = new Map<string, number>();
for (const message of messages) { for (const message of messages) {
const { rawMessageInfo } = message; const { rawMessageInfo } = message;
rawMessageInfos.push(rawMessageInfo); rawMessageInfos.push(rawMessageInfo);
const { threadID } = rawMessageInfo; const { threadID } = rawMessageInfo;
const currentCountValue = threadToMessageCount.get(threadID); const currentCountValue = threadToMessageCount.get(threadID);
const currentCount = currentCountValue ? currentCountValue : 0; const currentCount = currentCountValue ? currentCountValue : 0;
threadToMessageCount.set(threadID, currentCount + 1); threadToMessageCount.set(threadID, currentCount + 1);
} }
▲ Show 20 Lines • Show All 479 Lines • ▼ Show 20 Lines SELECT m.id, (
ORDER BY time DESC, id DESC ORDER BY time DESC, id DESC
LIMIT 1 LIMIT 1
) content ) content
FROM messages m FROM messages m
WHERE m.id IN(${messageIDs}) WHERE m.id IN(${messageIDs})
`; `;
const [result] = await dbQuery(latestEditedMessageQuery); const [result] = await dbQuery(latestEditedMessageQuery);
const latestContentByID = new Map(); const latestContentByID = new Map<string, EditMessageContent>();
for (const row of result) { for (const row of result) {
if (!row.content) { if (!row.content) {
continue; continue;
} }
const content = JSON.parse(row.content); const content = JSON.parse(row.content);
latestContentByID.set(row.id.toString(), content); latestContentByID.set(row.id.toString(), content);
} }
▲ Show 20 Lines • Show All 181 Lines • Show Last 20 Lines
View Options

services/blob/Cargo.lock

Show First 20 Lines • Show All 342 Lines • ▼ Show 20 Lines query.append(SQL`
ORDER BY tw.threadID, tw.time DESC, tw.id DESC ORDER BY tw.threadID, tw.time DESC, tw.id DESC
`); `);
const [result] = await dbQuery(query); const [result] = await dbQuery(query);
const derivedMessages = await fetchDerivedMessages(result, viewer); const derivedMessages = await fetchDerivedMessages(result, viewer);
const messages = await parseMessageSQLResult(result, derivedMessages, viewer); const messages = await parseMessageSQLResult(result, derivedMessages, viewer);
const rawMessageInfos = []; const rawMessageInfos = [];
const threadToMessageCount = new Map(); const threadToMessageCount = new Map<string, number>();
for (const message of messages) { for (const message of messages) {
const { rawMessageInfo } = message; const { rawMessageInfo } = message;
rawMessageInfos.push(rawMessageInfo); rawMessageInfos.push(rawMessageInfo);
const { threadID } = rawMessageInfo; const { threadID } = rawMessageInfo;
const currentCountValue = threadToMessageCount.get(threadID); const currentCountValue = threadToMessageCount.get(threadID);
const currentCount = currentCountValue ? currentCountValue : 0; const currentCount = currentCountValue ? currentCountValue : 0;
threadToMessageCount.set(threadID, currentCount + 1); threadToMessageCount.set(threadID, currentCount + 1);
} }
▲ Show 20 Lines • Show All 479 Lines • ▼ Show 20 Lines SELECT m.id, (
ORDER BY time DESC, id DESC ORDER BY time DESC, id DESC
LIMIT 1 LIMIT 1
) content ) content
FROM messages m FROM messages m
WHERE m.id IN(${messageIDs}) WHERE m.id IN(${messageIDs})
`; `;
const [result] = await dbQuery(latestEditedMessageQuery); const [result] = await dbQuery(latestEditedMessageQuery);
const latestContentByID = new Map(); const latestContentByID = new Map<string, EditMessageContent>();
for (const row of result) { for (const row of result) {
if (!row.content) { if (!row.content) {
continue; continue;
} }
const content = JSON.parse(row.content); const content = JSON.parse(row.content);
latestContentByID.set(row.id.toString(), content); latestContentByID.set(row.id.toString(), content);
} }
▲ Show 20 Lines • Show All 181 Lines • Show Last 20 Lines
View Options

services/comm-services-lib/Cargo.lock

Show First 20 Lines • Show All 342 Lines • ▼ Show 20 Lines query.append(SQL`
ORDER BY tw.threadID, tw.time DESC, tw.id DESC ORDER BY tw.threadID, tw.time DESC, tw.id DESC
`); `);
const [result] = await dbQuery(query); const [result] = await dbQuery(query);
const derivedMessages = await fetchDerivedMessages(result, viewer); const derivedMessages = await fetchDerivedMessages(result, viewer);
const messages = await parseMessageSQLResult(result, derivedMessages, viewer); const messages = await parseMessageSQLResult(result, derivedMessages, viewer);
const rawMessageInfos = []; const rawMessageInfos = [];
const threadToMessageCount = new Map(); const threadToMessageCount = new Map<string, number>();
for (const message of messages) { for (const message of messages) {
const { rawMessageInfo } = message; const { rawMessageInfo } = message;
rawMessageInfos.push(rawMessageInfo); rawMessageInfos.push(rawMessageInfo);
const { threadID } = rawMessageInfo; const { threadID } = rawMessageInfo;
const currentCountValue = threadToMessageCount.get(threadID); const currentCountValue = threadToMessageCount.get(threadID);
const currentCount = currentCountValue ? currentCountValue : 0; const currentCount = currentCountValue ? currentCountValue : 0;
threadToMessageCount.set(threadID, currentCount + 1); threadToMessageCount.set(threadID, currentCount + 1);
} }
▲ Show 20 Lines • Show All 479 Lines • ▼ Show 20 Lines SELECT m.id, (
ORDER BY time DESC, id DESC ORDER BY time DESC, id DESC
LIMIT 1 LIMIT 1
) content ) content
FROM messages m FROM messages m
WHERE m.id IN(${messageIDs}) WHERE m.id IN(${messageIDs})
`; `;
const [result] = await dbQuery(latestEditedMessageQuery); const [result] = await dbQuery(latestEditedMessageQuery);
const latestContentByID = new Map(); const latestContentByID = new Map<string, EditMessageContent>();
for (const row of result) { for (const row of result) {
if (!row.content) { if (!row.content) {
continue; continue;
} }
const content = JSON.parse(row.content); const content = JSON.parse(row.content);
latestContentByID.set(row.id.toString(), content); latestContentByID.set(row.id.toString(), content);
} }
▲ Show 20 Lines • Show All 181 Lines • Show Last 20 Lines
View Options

services/comm-services-lib/Cargo.toml

Show First 20 Lines • Show All 342 Lines • ▼ Show 20 Lines query.append(SQL`
ORDER BY tw.threadID, tw.time DESC, tw.id DESC ORDER BY tw.threadID, tw.time DESC, tw.id DESC
`); `);
const [result] = await dbQuery(query); const [result] = await dbQuery(query);
const derivedMessages = await fetchDerivedMessages(result, viewer); const derivedMessages = await fetchDerivedMessages(result, viewer);
const messages = await parseMessageSQLResult(result, derivedMessages, viewer); const messages = await parseMessageSQLResult(result, derivedMessages, viewer);
const rawMessageInfos = []; const rawMessageInfos = [];
const threadToMessageCount = new Map(); const threadToMessageCount = new Map<string, number>();
for (const message of messages) { for (const message of messages) {
const { rawMessageInfo } = message; const { rawMessageInfo } = message;
rawMessageInfos.push(rawMessageInfo); rawMessageInfos.push(rawMessageInfo);
const { threadID } = rawMessageInfo; const { threadID } = rawMessageInfo;
const currentCountValue = threadToMessageCount.get(threadID); const currentCountValue = threadToMessageCount.get(threadID);
const currentCount = currentCountValue ? currentCountValue : 0; const currentCount = currentCountValue ? currentCountValue : 0;
threadToMessageCount.set(threadID, currentCount + 1); threadToMessageCount.set(threadID, currentCount + 1);
} }
▲ Show 20 Lines • Show All 479 Lines • ▼ Show 20 Lines SELECT m.id, (
ORDER BY time DESC, id DESC ORDER BY time DESC, id DESC
LIMIT 1 LIMIT 1
) content ) content
FROM messages m FROM messages m
WHERE m.id IN(${messageIDs}) WHERE m.id IN(${messageIDs})
`; `;
const [result] = await dbQuery(latestEditedMessageQuery); const [result] = await dbQuery(latestEditedMessageQuery);
const latestContentByID = new Map(); const latestContentByID = new Map<string, EditMessageContent>();
for (const row of result) { for (const row of result) {
if (!row.content) { if (!row.content) {
continue; continue;
} }
const content = JSON.parse(row.content); const content = JSON.parse(row.content);
latestContentByID.set(row.id.toString(), content); latestContentByID.set(row.id.toString(), content);
} }
▲ Show 20 Lines • Show All 181 Lines • Show Last 20 Lines
View Options

services/comm-services-lib/src/blob/client.rs

Show First 20 Lines • Show All 342 Lines • ▼ Show 20 Lines query.append(SQL`
ORDER BY tw.threadID, tw.time DESC, tw.id DESC ORDER BY tw.threadID, tw.time DESC, tw.id DESC
`); `);
const [result] = await dbQuery(query); const [result] = await dbQuery(query);
const derivedMessages = await fetchDerivedMessages(result, viewer); const derivedMessages = await fetchDerivedMessages(result, viewer);
const messages = await parseMessageSQLResult(result, derivedMessages, viewer); const messages = await parseMessageSQLResult(result, derivedMessages, viewer);
const rawMessageInfos = []; const rawMessageInfos = [];
const threadToMessageCount = new Map(); const threadToMessageCount = new Map<string, number>();
for (const message of messages) { for (const message of messages) {
const { rawMessageInfo } = message; const { rawMessageInfo } = message;
rawMessageInfos.push(rawMessageInfo); rawMessageInfos.push(rawMessageInfo);
const { threadID } = rawMessageInfo; const { threadID } = rawMessageInfo;
const currentCountValue = threadToMessageCount.get(threadID); const currentCountValue = threadToMessageCount.get(threadID);
const currentCount = currentCountValue ? currentCountValue : 0; const currentCount = currentCountValue ? currentCountValue : 0;
threadToMessageCount.set(threadID, currentCount + 1); threadToMessageCount.set(threadID, currentCount + 1);
} }
▲ Show 20 Lines • Show All 479 Lines • ▼ Show 20 Lines SELECT m.id, (
ORDER BY time DESC, id DESC ORDER BY time DESC, id DESC
LIMIT 1 LIMIT 1
) content ) content
FROM messages m FROM messages m
WHERE m.id IN(${messageIDs}) WHERE m.id IN(${messageIDs})
`; `;
const [result] = await dbQuery(latestEditedMessageQuery); const [result] = await dbQuery(latestEditedMessageQuery);
const latestContentByID = new Map(); const latestContentByID = new Map<string, EditMessageContent>();
for (const row of result) { for (const row of result) {
if (!row.content) { if (!row.content) {
continue; continue;
} }
const content = JSON.parse(row.content); const content = JSON.parse(row.content);
latestContentByID.set(row.id.toString(), content); latestContentByID.set(row.id.toString(), content);
} }
▲ Show 20 Lines • Show All 181 Lines • Show Last 20 Lines
View Options

services/comm-services-lib/src/blob/mod.rs

Show First 20 Lines • Show All 342 Lines • ▼ Show 20 Lines query.append(SQL`
ORDER BY tw.threadID, tw.time DESC, tw.id DESC ORDER BY tw.threadID, tw.time DESC, tw.id DESC
`); `);
const [result] = await dbQuery(query); const [result] = await dbQuery(query);
const derivedMessages = await fetchDerivedMessages(result, viewer); const derivedMessages = await fetchDerivedMessages(result, viewer);
const messages = await parseMessageSQLResult(result, derivedMessages, viewer); const messages = await parseMessageSQLResult(result, derivedMessages, viewer);
const rawMessageInfos = []; const rawMessageInfos = [];
const threadToMessageCount = new Map(); const threadToMessageCount = new Map<string, number>();
for (const message of messages) { for (const message of messages) {
const { rawMessageInfo } = message; const { rawMessageInfo } = message;
rawMessageInfos.push(rawMessageInfo); rawMessageInfos.push(rawMessageInfo);
const { threadID } = rawMessageInfo; const { threadID } = rawMessageInfo;
const currentCountValue = threadToMessageCount.get(threadID); const currentCountValue = threadToMessageCount.get(threadID);
const currentCount = currentCountValue ? currentCountValue : 0; const currentCount = currentCountValue ? currentCountValue : 0;
threadToMessageCount.set(threadID, currentCount + 1); threadToMessageCount.set(threadID, currentCount + 1);
} }
▲ Show 20 Lines • Show All 479 Lines • ▼ Show 20 Lines SELECT m.id, (
ORDER BY time DESC, id DESC ORDER BY time DESC, id DESC
LIMIT 1 LIMIT 1
) content ) content
FROM messages m FROM messages m
WHERE m.id IN(${messageIDs}) WHERE m.id IN(${messageIDs})
`; `;
const [result] = await dbQuery(latestEditedMessageQuery); const [result] = await dbQuery(latestEditedMessageQuery);
const latestContentByID = new Map(); const latestContentByID = new Map<string, EditMessageContent>();
for (const row of result) { for (const row of result) {
if (!row.content) { if (!row.content) {
continue; continue;
} }
const content = JSON.parse(row.content); const content = JSON.parse(row.content);
latestContentByID.set(row.id.toString(), content); latestContentByID.set(row.id.toString(), content);
} }
▲ Show 20 Lines • Show All 181 Lines • Show Last 20 Lines
View Options

services/comm-services-lib/src/lib.rs

Loading...
View Options

services/feature-flags/Cargo.lock

Loading...