Page MenuHomePhabricator
Differential D12142

Implement JSI for EntryStore
ClosedPublic
Actions

Authored by marcin on May 21 2024, 6:14 AM.
Tags
None
Referenced Files
F6240447: D12142.diff
Thu, Apr 24, 10:49 PM
Unknown Object (File)
Tue, Apr 8, 1:44 PM
Unknown Object (File)
Tue, Apr 8, 1:10 PM
Unknown Object (File)
Tue, Apr 8, 12:43 PM
Unknown Object (File)
Tue, Apr 8, 12:04 PM
Unknown Object (File)
Tue, Apr 8, 10:13 AM
Unknown Object (File)
Tue, Apr 8, 2:24 AM
Unknown Object (File)
Mon, Apr 7, 10:59 PM
View All Files
Subscribers
ashoat

Details

Reviewers
tomek
kamil
Commits
rCOMM3c0246a0af6b: Implement JSI for EntryStore
Summary

This differential implements EntryStoreOperations and EntryStore

Test Plan

Compile

Diff Detail

Repository
rCOMM Comm
Lint
No Lint Coverage
Unit
No Test Coverage

Event Timeline

marcin created this revision.May 21 2024, 6:14 AM
Herald added a subscriber: ashoat. · View Herald TranscriptMay 21 2024, 6:14 AM
marcin added a parent revision: D12138: Implement SQLite API to handle entry store.May 21 2024, 6:15 AM
Harbormaster completed remote builds in B29092: Diff 40456.May 21 2024, 6:35 AM
marcin requested review of this revision.May 21 2024, 6:35 AM
marcin updated this revision to Diff 40464.May 21 2024, 6:55 AM

entry_info -> entry

Harbormaster failed remote builds in B29100: Diff 40464!May 21 2024, 7:32 AM
tomek accepted this revision.May 21 2024, 7:33 AM
tomek added inline comments.
native/cpp/CommonCpp/NativeModules/EntryStoreOperations.h
43 ↗(On Diff #40464)

newline

native/cpp/CommonCpp/NativeModules/PersistentStorageUtilities/DataStores/EntryStore.h
31 ↗(On Diff #40464)

newline

This revision is now accepted and ready to land.May 21 2024, 7:33 AM
marcin added a child revision: D12148: Connect pipes between C++ and JS for entry store.May 21 2024, 7:43 AM
marcin updated this revision to Diff 40475.May 21 2024, 7:47 AM

Rebase

marcin updated this revision to Diff 40479.May 21 2024, 8:02 AM

Fix build failure

Harbormaster failed remote builds in B29108: Diff 40475!May 21 2024, 8:22 AM
Harbormaster completed remote builds in B29112: Diff 40479.May 21 2024, 8:46 AM
marcin updated this revision to Diff 40569.May 22 2024, 8:40 AM

Add newlines

marcin edited the test plan for this revision. (Show Details)May 22 2024, 8:43 AM
Harbormaster completed remote builds in B29198: Diff 40569.May 22 2024, 9:45 AM
Closed by commit rCOMM3c0246a0af6b: Implement JSI for EntryStore (authored by marcin). · Explain WhyMay 27 2024, 5:45 AM
This revision was automatically updated to reflect the committed changes.
marcin added a commit: rCOMM3c0246a0af6b: Implement JSI for EntryStore.

Revision Contents
Changeset List

PathSize
native/
cpp/
CommonCpp/
NativeModules/
3 lines
2 lines
17 lines
42 lines
PersistentStorageUtilities/
DataStores/
30 lines
70 lines
ios/
Comm.xcodeproj/
8 lines

Diff 40569

View Options

native/cpp/CommonCpp/NativeModules/CMakeLists.txt

// TODO: get rid of this // TODO: get rid of this
#![allow(dead_code)] #![allow(dead_code)]
use std::collections::HashMap; use std::collections::HashMap;
use aws_sdk_dynamodb::model::AttributeValue; use aws_sdk_dynamodb::model::AttributeValue;
use chrono::{DateTime, Utc}; use chrono::{DateTime, Utc};
use tracing::warn;
use crate::{ use crate::{
constants::devices_table::*, constants::devices_table::*,
database::parse_string_attribute, database::parse_string_attribute,
error::{DBItemAttributeError, DBItemError, FromAttributeValue}, error::{DBItemAttributeError, DBItemError, FromAttributeValue},
grpc_services::protos::unauth::DeviceType, grpc_services::protos::unauth::DeviceType,
}; };
Show All 34 Lines
struct DeviceListKeyAttribute(DateTime<Utc>); struct DeviceListKeyAttribute(DateTime<Utc>);
impl From<DeviceIDAttribute> for AttributeValue { impl From<DeviceIDAttribute> for AttributeValue {
fn from(value: DeviceIDAttribute) -> Self { fn from(value: DeviceIDAttribute) -> Self {
AttributeValue::S(format!("{DEVICE_ITEM_KEY_PREFIX}{}", value.0)) AttributeValue::S(format!("{DEVICE_ITEM_KEY_PREFIX}{}", value.0))
} }
} }
impl From<DeviceListKeyAttribute> for AttributeValue {
fn from(value: DeviceListKeyAttribute) -> Self {
AttributeValue::S(format!(
"{DEVICE_LIST_KEY_PREFIX}{}",
value.0.to_rfc3339()
))
}
}
impl TryFrom<Option<AttributeValue>> for DeviceIDAttribute { impl TryFrom<Option<AttributeValue>> for DeviceIDAttribute {
type Error = DBItemError; type Error = DBItemError;
fn try_from(value: Option<AttributeValue>) -> Result<Self, Self::Error> { fn try_from(value: Option<AttributeValue>) -> Result<Self, Self::Error> {
let item_id = parse_string_attribute(ATTR_ITEM_ID, value)?; let item_id = parse_string_attribute(ATTR_ITEM_ID, value)?;
// remove the device- prefix // remove the device- prefix
let device_id = item_id let device_id = item_id
.strip_prefix(DEVICE_ITEM_KEY_PREFIX) .strip_prefix(DEVICE_ITEM_KEY_PREFIX)
.ok_or_else(|| DBItemError { .ok_or_else(|| DBItemError {
attribute_name: ATTR_ITEM_ID.to_string(), attribute_name: ATTR_ITEM_ID.to_string(),
attribute_value: Some(AttributeValue::S(item_id.clone())), attribute_value: Some(AttributeValue::S(item_id.clone())),
attribute_error: DBItemAttributeError::InvalidValue, attribute_error: DBItemAttributeError::InvalidValue,
})? })?
.to_string(); .to_string();
Ok(Self(device_id)) Ok(Self(device_id))
} }
} }
impl TryFrom<Option<AttributeValue>> for DeviceListKeyAttribute {
type Error = DBItemError;
fn try_from(value: Option<AttributeValue>) -> Result<Self, Self::Error> {
let item_id = parse_string_attribute(ATTR_ITEM_ID, value)?;
// remove the device-list- prefix, then parse the timestamp
let timestamp: DateTime<Utc> = item_id
.strip_prefix(DEVICE_LIST_KEY_PREFIX)
.ok_or_else(|| DBItemError {
attribute_name: ATTR_ITEM_ID.to_string(),
attribute_value: Some(AttributeValue::S(item_id.clone())),
attribute_error: DBItemAttributeError::InvalidValue,
})
.and_then(|s| {
s.parse().map_err(|e| {
DBItemError::new(
ATTR_ITEM_ID.to_string(),
Some(AttributeValue::S(item_id.clone())),
DBItemAttributeError::InvalidTimestamp(e),
)
})
})?;
Ok(Self(timestamp))
}
}
impl TryFrom<RawAttributes> for DeviceRow { impl TryFrom<RawAttributes> for DeviceRow {
type Error = DBItemError; type Error = DBItemError;
fn try_from(mut attrs: RawAttributes) -> Result<Self, Self::Error> { fn try_from(mut attrs: RawAttributes) -> Result<Self, Self::Error> {
let user_id = let user_id =
parse_string_attribute(ATTR_USER_ID, attrs.remove(ATTR_USER_ID))?; parse_string_attribute(ATTR_USER_ID, attrs.remove(ATTR_USER_ID))?;
let DeviceIDAttribute(device_id) = attrs.remove(ATTR_ITEM_ID).try_into()?; let DeviceIDAttribute(device_id) = attrs.remove(ATTR_ITEM_ID).try_into()?;
▲ Show 20 Lines • Show All 95 Lines • ▼ Show 20 Lines if let Some(social_proof) = value.social_proof {
ATTR_SOCIAL_PROOF.to_string(), ATTR_SOCIAL_PROOF.to_string(),
AttributeValue::S(social_proof), AttributeValue::S(social_proof),
); );
} }
attrs attrs
} }
} }
impl TryFrom<RawAttributes> for DeviceListRow {
type Error = DBItemError;
fn try_from(mut attrs: RawAttributes) -> Result<Self, Self::Error> {
let user_id =
parse_string_attribute(ATTR_USER_ID, attrs.remove(ATTR_USER_ID))?;
let DeviceListKeyAttribute(timestamp) =
attrs.remove(ATTR_ITEM_ID).try_into()?;
// validate timestamps are in sync
let timestamps_match = attrs
.remove(ATTR_TIMESTAMP)
.and_then(|attr| attr.as_n().ok().cloned())
.and_then(|val| val.parse::<i64>().ok())
.filter(|val| *val == timestamp.timestamp_millis())
.is_some();
if !timestamps_match {
warn!(
"DeviceList timestamp mismatch for (userID={}, itemID={})",
&user_id,
timestamp.to_rfc3339()
);
}
// this should be a list of strings
let device_ids = attrs
.remove(ATTR_DEVICE_IDS)
.ok_or_else(|| {
DBItemError::new(
ATTR_DEVICE_IDS.to_string(),
None,
DBItemAttributeError::Missing,
)
})?
.to_vec(ATTR_DEVICE_IDS)?
.iter()
.map(|v| v.to_string("device_ids[?]").cloned())
.collect::<Result<Vec<String>, DBItemError>>()?;
Ok(Self {
user_id,
timestamp,
device_ids,
})
}
}
impl From<DeviceListRow> for RawAttributes {
fn from(device_list: DeviceListRow) -> Self {
let mut attrs = HashMap::new();
attrs.insert(
ATTR_USER_ID.to_string(),
AttributeValue::S(device_list.user_id.clone()),
);
attrs.insert(
ATTR_ITEM_ID.to_string(),
DeviceListKeyAttribute(device_list.timestamp).into(),
);
attrs.insert(
ATTR_TIMESTAMP.to_string(),
AttributeValue::N(device_list.timestamp.timestamp_millis().to_string()),
);
bartekAuthorUnsubmitted
Done
Inline Actions

Note that we use RFC 3339 string for ItemID while the ATTR_TIMESTAMP uses milliseconds number. Generally I use the former everywhere (simpler parsing without NaiveDateTime conversions) but I thought the latter is easier for DDB to sort by (it's used only as index sorting key and not read anywhere else)

bartek: Note that we use RFC 3339 string for `ItemID` while the `ATTR_TIMESTAMP` uses milliseconds…
attrs.insert(
ATTR_DEVICE_IDS.to_string(),
AttributeValue::L(
device_list
.device_ids
.into_iter()
.map(AttributeValue::S)
.collect(),
),
);
attrs
}
}
View Options

native/cpp/CommonCpp/NativeModules/CommCoreModule.h

// TODO: get rid of this // TODO: get rid of this
#![allow(dead_code)] #![allow(dead_code)]
use std::collections::HashMap; use std::collections::HashMap;
use aws_sdk_dynamodb::model::AttributeValue; use aws_sdk_dynamodb::model::AttributeValue;
use chrono::{DateTime, Utc}; use chrono::{DateTime, Utc};
use tracing::warn;
use crate::{ use crate::{
constants::devices_table::*, constants::devices_table::*,
database::parse_string_attribute, database::parse_string_attribute,
error::{DBItemAttributeError, DBItemError, FromAttributeValue}, error::{DBItemAttributeError, DBItemError, FromAttributeValue},
grpc_services::protos::unauth::DeviceType, grpc_services::protos::unauth::DeviceType,
}; };
Show All 34 Lines
struct DeviceListKeyAttribute(DateTime<Utc>); struct DeviceListKeyAttribute(DateTime<Utc>);
impl From<DeviceIDAttribute> for AttributeValue { impl From<DeviceIDAttribute> for AttributeValue {
fn from(value: DeviceIDAttribute) -> Self { fn from(value: DeviceIDAttribute) -> Self {
AttributeValue::S(format!("{DEVICE_ITEM_KEY_PREFIX}{}", value.0)) AttributeValue::S(format!("{DEVICE_ITEM_KEY_PREFIX}{}", value.0))
} }
} }
impl From<DeviceListKeyAttribute> for AttributeValue {
fn from(value: DeviceListKeyAttribute) -> Self {
AttributeValue::S(format!(
"{DEVICE_LIST_KEY_PREFIX}{}",
value.0.to_rfc3339()
))
}
}
impl TryFrom<Option<AttributeValue>> for DeviceIDAttribute { impl TryFrom<Option<AttributeValue>> for DeviceIDAttribute {
type Error = DBItemError; type Error = DBItemError;
fn try_from(value: Option<AttributeValue>) -> Result<Self, Self::Error> { fn try_from(value: Option<AttributeValue>) -> Result<Self, Self::Error> {
let item_id = parse_string_attribute(ATTR_ITEM_ID, value)?; let item_id = parse_string_attribute(ATTR_ITEM_ID, value)?;
// remove the device- prefix // remove the device- prefix
let device_id = item_id let device_id = item_id
.strip_prefix(DEVICE_ITEM_KEY_PREFIX) .strip_prefix(DEVICE_ITEM_KEY_PREFIX)
.ok_or_else(|| DBItemError { .ok_or_else(|| DBItemError {
attribute_name: ATTR_ITEM_ID.to_string(), attribute_name: ATTR_ITEM_ID.to_string(),
attribute_value: Some(AttributeValue::S(item_id.clone())), attribute_value: Some(AttributeValue::S(item_id.clone())),
attribute_error: DBItemAttributeError::InvalidValue, attribute_error: DBItemAttributeError::InvalidValue,
})? })?
.to_string(); .to_string();
Ok(Self(device_id)) Ok(Self(device_id))
} }
} }
impl TryFrom<Option<AttributeValue>> for DeviceListKeyAttribute {
type Error = DBItemError;
fn try_from(value: Option<AttributeValue>) -> Result<Self, Self::Error> {
let item_id = parse_string_attribute(ATTR_ITEM_ID, value)?;
// remove the device-list- prefix, then parse the timestamp
let timestamp: DateTime<Utc> = item_id
.strip_prefix(DEVICE_LIST_KEY_PREFIX)
.ok_or_else(|| DBItemError {
attribute_name: ATTR_ITEM_ID.to_string(),
attribute_value: Some(AttributeValue::S(item_id.clone())),
attribute_error: DBItemAttributeError::InvalidValue,
})
.and_then(|s| {
s.parse().map_err(|e| {
DBItemError::new(
ATTR_ITEM_ID.to_string(),
Some(AttributeValue::S(item_id.clone())),
DBItemAttributeError::InvalidTimestamp(e),
)
})
})?;
Ok(Self(timestamp))
}
}
impl TryFrom<RawAttributes> for DeviceRow { impl TryFrom<RawAttributes> for DeviceRow {
type Error = DBItemError; type Error = DBItemError;
fn try_from(mut attrs: RawAttributes) -> Result<Self, Self::Error> { fn try_from(mut attrs: RawAttributes) -> Result<Self, Self::Error> {
let user_id = let user_id =
parse_string_attribute(ATTR_USER_ID, attrs.remove(ATTR_USER_ID))?; parse_string_attribute(ATTR_USER_ID, attrs.remove(ATTR_USER_ID))?;
let DeviceIDAttribute(device_id) = attrs.remove(ATTR_ITEM_ID).try_into()?; let DeviceIDAttribute(device_id) = attrs.remove(ATTR_ITEM_ID).try_into()?;
▲ Show 20 Lines • Show All 95 Lines • ▼ Show 20 Lines if let Some(social_proof) = value.social_proof {
ATTR_SOCIAL_PROOF.to_string(), ATTR_SOCIAL_PROOF.to_string(),
AttributeValue::S(social_proof), AttributeValue::S(social_proof),
); );
} }
attrs attrs
} }
} }
impl TryFrom<RawAttributes> for DeviceListRow {
type Error = DBItemError;
fn try_from(mut attrs: RawAttributes) -> Result<Self, Self::Error> {
let user_id =
parse_string_attribute(ATTR_USER_ID, attrs.remove(ATTR_USER_ID))?;
let DeviceListKeyAttribute(timestamp) =
attrs.remove(ATTR_ITEM_ID).try_into()?;
// validate timestamps are in sync
let timestamps_match = attrs
.remove(ATTR_TIMESTAMP)
.and_then(|attr| attr.as_n().ok().cloned())
.and_then(|val| val.parse::<i64>().ok())
.filter(|val| *val == timestamp.timestamp_millis())
.is_some();
if !timestamps_match {
warn!(
"DeviceList timestamp mismatch for (userID={}, itemID={})",
&user_id,
timestamp.to_rfc3339()
);
}
// this should be a list of strings
let device_ids = attrs
.remove(ATTR_DEVICE_IDS)
.ok_or_else(|| {
DBItemError::new(
ATTR_DEVICE_IDS.to_string(),
None,
DBItemAttributeError::Missing,
)
})?
.to_vec(ATTR_DEVICE_IDS)?
.iter()
.map(|v| v.to_string("device_ids[?]").cloned())
.collect::<Result<Vec<String>, DBItemError>>()?;
Ok(Self {
user_id,
timestamp,
device_ids,
})
}
}
impl From<DeviceListRow> for RawAttributes {
fn from(device_list: DeviceListRow) -> Self {
let mut attrs = HashMap::new();
attrs.insert(
ATTR_USER_ID.to_string(),
AttributeValue::S(device_list.user_id.clone()),
);
attrs.insert(
ATTR_ITEM_ID.to_string(),
DeviceListKeyAttribute(device_list.timestamp).into(),
);
attrs.insert(
ATTR_TIMESTAMP.to_string(),
AttributeValue::N(device_list.timestamp.timestamp_millis().to_string()),
);
bartekAuthorUnsubmitted
Done
Inline Actions

Note that we use RFC 3339 string for ItemID while the ATTR_TIMESTAMP uses milliseconds number. Generally I use the former everywhere (simpler parsing without NaiveDateTime conversions) but I thought the latter is easier for DDB to sort by (it's used only as index sorting key and not read anywhere else)

bartek: Note that we use RFC 3339 string for `ItemID` while the `ATTR_TIMESTAMP` uses milliseconds…
attrs.insert(
ATTR_DEVICE_IDS.to_string(),
AttributeValue::L(
device_list
.device_ids
.into_iter()
.map(AttributeValue::S)
.collect(),
),
);
attrs
}
}
View Options

native/cpp/CommonCpp/NativeModules/CommCoreModule.cpp

// TODO: get rid of this // TODO: get rid of this
#![allow(dead_code)] #![allow(dead_code)]
use std::collections::HashMap; use std::collections::HashMap;
use aws_sdk_dynamodb::model::AttributeValue; use aws_sdk_dynamodb::model::AttributeValue;
use chrono::{DateTime, Utc}; use chrono::{DateTime, Utc};
use tracing::warn;
use crate::{ use crate::{
constants::devices_table::*, constants::devices_table::*,
database::parse_string_attribute, database::parse_string_attribute,
error::{DBItemAttributeError, DBItemError, FromAttributeValue}, error::{DBItemAttributeError, DBItemError, FromAttributeValue},
grpc_services::protos::unauth::DeviceType, grpc_services::protos::unauth::DeviceType,
}; };
Show All 34 Lines
struct DeviceListKeyAttribute(DateTime<Utc>); struct DeviceListKeyAttribute(DateTime<Utc>);
impl From<DeviceIDAttribute> for AttributeValue { impl From<DeviceIDAttribute> for AttributeValue {
fn from(value: DeviceIDAttribute) -> Self { fn from(value: DeviceIDAttribute) -> Self {
AttributeValue::S(format!("{DEVICE_ITEM_KEY_PREFIX}{}", value.0)) AttributeValue::S(format!("{DEVICE_ITEM_KEY_PREFIX}{}", value.0))
} }
} }
impl From<DeviceListKeyAttribute> for AttributeValue {
fn from(value: DeviceListKeyAttribute) -> Self {
AttributeValue::S(format!(
"{DEVICE_LIST_KEY_PREFIX}{}",
value.0.to_rfc3339()
))
}
}
impl TryFrom<Option<AttributeValue>> for DeviceIDAttribute { impl TryFrom<Option<AttributeValue>> for DeviceIDAttribute {
type Error = DBItemError; type Error = DBItemError;
fn try_from(value: Option<AttributeValue>) -> Result<Self, Self::Error> { fn try_from(value: Option<AttributeValue>) -> Result<Self, Self::Error> {
let item_id = parse_string_attribute(ATTR_ITEM_ID, value)?; let item_id = parse_string_attribute(ATTR_ITEM_ID, value)?;
// remove the device- prefix // remove the device- prefix
let device_id = item_id let device_id = item_id
.strip_prefix(DEVICE_ITEM_KEY_PREFIX) .strip_prefix(DEVICE_ITEM_KEY_PREFIX)
.ok_or_else(|| DBItemError { .ok_or_else(|| DBItemError {
attribute_name: ATTR_ITEM_ID.to_string(), attribute_name: ATTR_ITEM_ID.to_string(),
attribute_value: Some(AttributeValue::S(item_id.clone())), attribute_value: Some(AttributeValue::S(item_id.clone())),
attribute_error: DBItemAttributeError::InvalidValue, attribute_error: DBItemAttributeError::InvalidValue,
})? })?
.to_string(); .to_string();
Ok(Self(device_id)) Ok(Self(device_id))
} }
} }
impl TryFrom<Option<AttributeValue>> for DeviceListKeyAttribute {
type Error = DBItemError;
fn try_from(value: Option<AttributeValue>) -> Result<Self, Self::Error> {
let item_id = parse_string_attribute(ATTR_ITEM_ID, value)?;
// remove the device-list- prefix, then parse the timestamp
let timestamp: DateTime<Utc> = item_id
.strip_prefix(DEVICE_LIST_KEY_PREFIX)
.ok_or_else(|| DBItemError {
attribute_name: ATTR_ITEM_ID.to_string(),
attribute_value: Some(AttributeValue::S(item_id.clone())),
attribute_error: DBItemAttributeError::InvalidValue,
})
.and_then(|s| {
s.parse().map_err(|e| {
DBItemError::new(
ATTR_ITEM_ID.to_string(),
Some(AttributeValue::S(item_id.clone())),
DBItemAttributeError::InvalidTimestamp(e),
)
})
})?;
Ok(Self(timestamp))
}
}
impl TryFrom<RawAttributes> for DeviceRow { impl TryFrom<RawAttributes> for DeviceRow {
type Error = DBItemError; type Error = DBItemError;
fn try_from(mut attrs: RawAttributes) -> Result<Self, Self::Error> { fn try_from(mut attrs: RawAttributes) -> Result<Self, Self::Error> {
let user_id = let user_id =
parse_string_attribute(ATTR_USER_ID, attrs.remove(ATTR_USER_ID))?; parse_string_attribute(ATTR_USER_ID, attrs.remove(ATTR_USER_ID))?;
let DeviceIDAttribute(device_id) = attrs.remove(ATTR_ITEM_ID).try_into()?; let DeviceIDAttribute(device_id) = attrs.remove(ATTR_ITEM_ID).try_into()?;
▲ Show 20 Lines • Show All 95 Lines • ▼ Show 20 Lines if let Some(social_proof) = value.social_proof {
ATTR_SOCIAL_PROOF.to_string(), ATTR_SOCIAL_PROOF.to_string(),
AttributeValue::S(social_proof), AttributeValue::S(social_proof),
); );
} }
attrs attrs
} }
} }
impl TryFrom<RawAttributes> for DeviceListRow {
type Error = DBItemError;
fn try_from(mut attrs: RawAttributes) -> Result<Self, Self::Error> {
let user_id =
parse_string_attribute(ATTR_USER_ID, attrs.remove(ATTR_USER_ID))?;
let DeviceListKeyAttribute(timestamp) =
attrs.remove(ATTR_ITEM_ID).try_into()?;
// validate timestamps are in sync
let timestamps_match = attrs
.remove(ATTR_TIMESTAMP)
.and_then(|attr| attr.as_n().ok().cloned())
.and_then(|val| val.parse::<i64>().ok())
.filter(|val| *val == timestamp.timestamp_millis())
.is_some();
if !timestamps_match {
warn!(
"DeviceList timestamp mismatch for (userID={}, itemID={})",
&user_id,
timestamp.to_rfc3339()
);
}
// this should be a list of strings
let device_ids = attrs
.remove(ATTR_DEVICE_IDS)
.ok_or_else(|| {
DBItemError::new(
ATTR_DEVICE_IDS.to_string(),
None,
DBItemAttributeError::Missing,
)
})?
.to_vec(ATTR_DEVICE_IDS)?
.iter()
.map(|v| v.to_string("device_ids[?]").cloned())
.collect::<Result<Vec<String>, DBItemError>>()?;
Ok(Self {
user_id,
timestamp,
device_ids,
})
}
}
impl From<DeviceListRow> for RawAttributes {
fn from(device_list: DeviceListRow) -> Self {
let mut attrs = HashMap::new();
attrs.insert(
ATTR_USER_ID.to_string(),
AttributeValue::S(device_list.user_id.clone()),
);
attrs.insert(
ATTR_ITEM_ID.to_string(),
DeviceListKeyAttribute(device_list.timestamp).into(),
);
attrs.insert(
ATTR_TIMESTAMP.to_string(),
AttributeValue::N(device_list.timestamp.timestamp_millis().to_string()),
);
bartekAuthorUnsubmitted
Done
Inline Actions

Note that we use RFC 3339 string for ItemID while the ATTR_TIMESTAMP uses milliseconds number. Generally I use the former everywhere (simpler parsing without NaiveDateTime conversions) but I thought the latter is easier for DDB to sort by (it's used only as index sorting key and not read anywhere else)

bartek: Note that we use RFC 3339 string for `ItemID` while the `ATTR_TIMESTAMP` uses milliseconds…
attrs.insert(
ATTR_DEVICE_IDS.to_string(),
AttributeValue::L(
device_list
.device_ids
.into_iter()
.map(AttributeValue::S)
.collect(),
),
);
attrs
}
}
View Options

native/cpp/CommonCpp/NativeModules/EntryStoreOperations.h

// TODO: get rid of this // TODO: get rid of this
#![allow(dead_code)] #![allow(dead_code)]
use std::collections::HashMap; use std::collections::HashMap;
use aws_sdk_dynamodb::model::AttributeValue; use aws_sdk_dynamodb::model::AttributeValue;
use chrono::{DateTime, Utc}; use chrono::{DateTime, Utc};
use tracing::warn;
use crate::{ use crate::{
constants::devices_table::*, constants::devices_table::*,
database::parse_string_attribute, database::parse_string_attribute,
error::{DBItemAttributeError, DBItemError, FromAttributeValue}, error::{DBItemAttributeError, DBItemError, FromAttributeValue},
grpc_services::protos::unauth::DeviceType, grpc_services::protos::unauth::DeviceType,
}; };
Show All 34 Lines
struct DeviceListKeyAttribute(DateTime<Utc>); struct DeviceListKeyAttribute(DateTime<Utc>);
impl From<DeviceIDAttribute> for AttributeValue { impl From<DeviceIDAttribute> for AttributeValue {
fn from(value: DeviceIDAttribute) -> Self { fn from(value: DeviceIDAttribute) -> Self {
AttributeValue::S(format!("{DEVICE_ITEM_KEY_PREFIX}{}", value.0)) AttributeValue::S(format!("{DEVICE_ITEM_KEY_PREFIX}{}", value.0))
} }
} }
impl From<DeviceListKeyAttribute> for AttributeValue {
fn from(value: DeviceListKeyAttribute) -> Self {
AttributeValue::S(format!(
"{DEVICE_LIST_KEY_PREFIX}{}",
value.0.to_rfc3339()
))
}
}
impl TryFrom<Option<AttributeValue>> for DeviceIDAttribute { impl TryFrom<Option<AttributeValue>> for DeviceIDAttribute {
type Error = DBItemError; type Error = DBItemError;
fn try_from(value: Option<AttributeValue>) -> Result<Self, Self::Error> { fn try_from(value: Option<AttributeValue>) -> Result<Self, Self::Error> {
let item_id = parse_string_attribute(ATTR_ITEM_ID, value)?; let item_id = parse_string_attribute(ATTR_ITEM_ID, value)?;
// remove the device- prefix // remove the device- prefix
let device_id = item_id let device_id = item_id
.strip_prefix(DEVICE_ITEM_KEY_PREFIX) .strip_prefix(DEVICE_ITEM_KEY_PREFIX)
.ok_or_else(|| DBItemError { .ok_or_else(|| DBItemError {
attribute_name: ATTR_ITEM_ID.to_string(), attribute_name: ATTR_ITEM_ID.to_string(),
attribute_value: Some(AttributeValue::S(item_id.clone())), attribute_value: Some(AttributeValue::S(item_id.clone())),
attribute_error: DBItemAttributeError::InvalidValue, attribute_error: DBItemAttributeError::InvalidValue,
})? })?
.to_string(); .to_string();
Ok(Self(device_id)) Ok(Self(device_id))
} }
} }
impl TryFrom<Option<AttributeValue>> for DeviceListKeyAttribute {
type Error = DBItemError;
fn try_from(value: Option<AttributeValue>) -> Result<Self, Self::Error> {
let item_id = parse_string_attribute(ATTR_ITEM_ID, value)?;
// remove the device-list- prefix, then parse the timestamp
let timestamp: DateTime<Utc> = item_id
.strip_prefix(DEVICE_LIST_KEY_PREFIX)
.ok_or_else(|| DBItemError {
attribute_name: ATTR_ITEM_ID.to_string(),
attribute_value: Some(AttributeValue::S(item_id.clone())),
attribute_error: DBItemAttributeError::InvalidValue,
})
.and_then(|s| {
s.parse().map_err(|e| {
DBItemError::new(
ATTR_ITEM_ID.to_string(),
Some(AttributeValue::S(item_id.clone())),
DBItemAttributeError::InvalidTimestamp(e),
)
})
})?;
Ok(Self(timestamp))
}
}
impl TryFrom<RawAttributes> for DeviceRow { impl TryFrom<RawAttributes> for DeviceRow {
type Error = DBItemError; type Error = DBItemError;
fn try_from(mut attrs: RawAttributes) -> Result<Self, Self::Error> { fn try_from(mut attrs: RawAttributes) -> Result<Self, Self::Error> {
let user_id = let user_id =
parse_string_attribute(ATTR_USER_ID, attrs.remove(ATTR_USER_ID))?; parse_string_attribute(ATTR_USER_ID, attrs.remove(ATTR_USER_ID))?;
let DeviceIDAttribute(device_id) = attrs.remove(ATTR_ITEM_ID).try_into()?; let DeviceIDAttribute(device_id) = attrs.remove(ATTR_ITEM_ID).try_into()?;
▲ Show 20 Lines • Show All 95 Lines • ▼ Show 20 Lines if let Some(social_proof) = value.social_proof {
ATTR_SOCIAL_PROOF.to_string(), ATTR_SOCIAL_PROOF.to_string(),
AttributeValue::S(social_proof), AttributeValue::S(social_proof),
); );
} }
attrs attrs
} }
} }
impl TryFrom<RawAttributes> for DeviceListRow {
type Error = DBItemError;
fn try_from(mut attrs: RawAttributes) -> Result<Self, Self::Error> {
let user_id =
parse_string_attribute(ATTR_USER_ID, attrs.remove(ATTR_USER_ID))?;
let DeviceListKeyAttribute(timestamp) =
attrs.remove(ATTR_ITEM_ID).try_into()?;
// validate timestamps are in sync
let timestamps_match = attrs
.remove(ATTR_TIMESTAMP)
.and_then(|attr| attr.as_n().ok().cloned())
.and_then(|val| val.parse::<i64>().ok())
.filter(|val| *val == timestamp.timestamp_millis())
.is_some();
if !timestamps_match {
warn!(
"DeviceList timestamp mismatch for (userID={}, itemID={})",
&user_id,
timestamp.to_rfc3339()
);
}
// this should be a list of strings
let device_ids = attrs
.remove(ATTR_DEVICE_IDS)
.ok_or_else(|| {
DBItemError::new(
ATTR_DEVICE_IDS.to_string(),
None,
DBItemAttributeError::Missing,
)
})?
.to_vec(ATTR_DEVICE_IDS)?
.iter()
.map(|v| v.to_string("device_ids[?]").cloned())
.collect::<Result<Vec<String>, DBItemError>>()?;
Ok(Self {
user_id,
timestamp,
device_ids,
})
}
}
impl From<DeviceListRow> for RawAttributes {
fn from(device_list: DeviceListRow) -> Self {
let mut attrs = HashMap::new();
attrs.insert(
ATTR_USER_ID.to_string(),
AttributeValue::S(device_list.user_id.clone()),
);
attrs.insert(
ATTR_ITEM_ID.to_string(),
DeviceListKeyAttribute(device_list.timestamp).into(),
);
attrs.insert(
ATTR_TIMESTAMP.to_string(),
AttributeValue::N(device_list.timestamp.timestamp_millis().to_string()),
);
bartekAuthorUnsubmitted
Done
Inline Actions

Note that we use RFC 3339 string for ItemID while the ATTR_TIMESTAMP uses milliseconds number. Generally I use the former everywhere (simpler parsing without NaiveDateTime conversions) but I thought the latter is easier for DDB to sort by (it's used only as index sorting key and not read anywhere else)

bartek: Note that we use RFC 3339 string for `ItemID` while the `ATTR_TIMESTAMP` uses milliseconds…
attrs.insert(
ATTR_DEVICE_IDS.to_string(),
AttributeValue::L(
device_list
.device_ids
.into_iter()
.map(AttributeValue::S)
.collect(),
),
);
attrs
}
}
View Options

native/cpp/CommonCpp/NativeModules/PersistentStorageUtilities/DataStores/EntryStore.h

Loading...
View Options

native/cpp/CommonCpp/NativeModules/PersistentStorageUtilities/DataStores/EntryStore.cpp

Loading...
View Options

native/ios/Comm.xcodeproj/project.pbxproj

Loading...