Page MenuHomePhabricator

Implement translation of message JSON object into SQLite database entities
ClosedPublic

Authored by marcin on May 10 2022, 2:50 AM.
Tags
None
Referenced Files
Unknown Object (File)
Fri, Dec 20, 2:03 AM
Unknown Object (File)
Fri, Dec 20, 2:02 AM
Unknown Object (File)
Fri, Dec 20, 2:02 AM
Unknown Object (File)
Fri, Dec 20, 1:02 AM
Unknown Object (File)
Fri, Dec 20, 1:01 AM
Unknown Object (File)
Thu, Dec 19, 2:17 PM
Unknown Object (File)
Tue, Dec 17, 4:39 PM
Unknown Object (File)
Tue, Dec 17, 4:39 PM

Details

Summary

Implement function that translates JSON object representing notification content into Message and Media database entities

Test Plan

No testing plan. Future diffs will provide a code that uses those functions on iOS and Android respectively

Diff Detail

Repository
rCOMM Comm
Lint
Lint Not Applicable
Unit
Tests Not Applicable

Event Timeline

Update commit message to include diff link

native/cpp/CommonCpp/NativeModules/MessageOperationsUtilities/MessageOperationsUtilities.cpp
35 ↗(On Diff #12486)

Before splitting D3941 into smaller diffs @karol-bisztyga asked an important question:

How does this work with folly::dynamic? Don't we need to check whether it is a collection and if the field is present in the first place?

We can always iterate in foreach manner over folly::dynamic provided it is an Object or an Array. In the first case the iteration works as iteration over dictionary (key-value) pairs and as iteration over vector in the the case on an Array. I understand there are several places in this code where I assume that a certain key is present or the value under certain key is an Object or an Array. On one hand it would be justifiable to add sanity checks in those places and throw some error/log and return if the condition is not fulfilled. On the other this code is meant to mirror functionality implemented in message-ops-utils.js where assumptions about a presence of a certain key or character ot the value are not additionally verified. Therefore I assumed it might be an overkill to introduce such sanity checks in C++ version of the code. However it is not an issue to provide such checks. @ashoat, @palys-swm what do you think? Can I assume that if presence of a key or value character in notification is not verified in JS then it does not need to be verified in C++?

native/cpp/CommonCpp/NativeModules/MessageOperationsUtilities/MessageOperationsUtilities.cpp
35 ↗(On Diff #12486)

(I responded on D3941 because I saw that first... here is my response copy-pasted:)

On the other this code is meant to mirror functionality implemented in message-ops-utils.js where assumptions about a presence of a certain key or character ot the value are not additionally verified.

In our JS codebase, I think these assumptions are generally enforced statically by the Flow typechecker. But I don't think data returned from the server is verified at runtime, so the assumptions made by the typechecker could be wrong.

I think it's probably okay to make these assumptions since we do the same in JS, but I guess it would be better to do some sort of runtime checks when the data is received from the server. I guess that's probably outside the scope of this diff?

I honestly don't feel too strongly, open to other perspectives.

I'd add these checks. If they're not present in the js code then either there's a solid reason for it or the code just misses them.

As a side note - maybe it's good to check what exactly happens when we try to access and perform operations on an object from the dynamic collection and it's not present. We should be aware of what may happen. Do we get nullptr? Or do we get a UB with a random value from the memory? Is there an error thrown? Maybe something else?

This revision now requires changes to proceed.May 10 2022, 6:18 AM
native/cpp/CommonCpp/NativeModules/MessageOperationsUtilities/MessageOperationsUtilities.cpp
35 ↗(On Diff #12486)

I wouldn't say it is out of the scope of this diff. Instead of optimistically accessing keys we could use if statements or ternary operators to check for presence of given key or whether value we expect under the given key is indeed and Object or an Array so that it can be iterated over. The only difficulty I see in performing runtime checks is how should we react when those checks fail? Probably we should throw an error or log error message and return, but how do we make sure that corrupted notification is re-send in a correct form, or is at least downloaded in its correct form upon app launch?

native/cpp/CommonCpp/NativeModules/MessageOperationsUtilities/MessageOperationsUtilities.cpp
35 ↗(On Diff #12486)

I think we can skip processing the corrupted notif if it is malformed. Since we don't advance messagesCurrentAsOf in response to messages saved via notif, we will still end up downloading the same message using the standard in-app mechanism. So if we skip processing a corrupted notif, it should have a minimal impact.

Of course, it's likely that if the RawMessageInfo in the notif is corrupted, it will probably be corrupted when downloaded via the standard in-app mechanism.

The main exception I can think of – what happens if the RawMessageInfo is really long? Is it possible that it gets truncated by the notif service (APNs or FCM)? We should probably make sure we don't crash if the JSON parse fails

Today I was thinking about error handling and had a brief discussion with @palys-swm about error handling and here are my findings and conclusions:
We have two types of errors:

  • parsing string into folly::dynamic fails
  • parsing folly::dynamic into Message and Media entities fails

The first case is neither a part of this diff nor its parent since both introduce functions that operate on folly::dynamic. Subsequent diff introduces method that operates on std::string so error handling in this case will be discussed later.
In the second case errors can result from either:

  • Missing keys in JSON object - this throws std::out_of_bound error
  • Wrong types under present keys - e.g. there is an int under given key but we expect to receive an array - this throws folly::TypeError

I established with @palys-swm that any error in message structure should prevent us from persisting it in database - e.g. we do not want to persist message with valid thread, id, user itp, but invalid content (we could store content as nullptr) nor multimedia message with correct message structure but invalid media JSON objects. So I suggest that this method and the one introduced in parent diff are made private and do not perform error handling. Error handling will be performed in the next diff that introduces a method that uses these two to parse raw string (translateStringTOClientDBMessageInfos in D3941). There we can simply try() ... catch() and skip the message with proper error logs. The reason ti make translateRawMessageInfoToClientDBMessageInfo and translateMediaToClientDBMediaInfo private is that no one uses them outside of MessageOperationsUtilities public method where error handling will take place.

tomek requested changes to this revision.May 11 2022, 4:36 AM
tomek added inline comments.
native/cpp/CommonCpp/NativeModules/MessageOperationsUtilities/MessageOperationsUtilities.cpp
12 ↗(On Diff #12486)

What is the reason behind using count instead of find?

23–26 ↗(On Diff #12486)

What do you think about using if instead?

41–49 ↗(On Diff #12486)

Why do we move some fields but not all of them?

native/cpp/CommonCpp/NativeModules/MessageOperationsUtilities/MessageOperationsUtilities.cpp
41–49 ↗(On Diff #12486)

It looks like inconsistency I accidentally made. fields that are std::string should not be moved. We should only move fields that are std::unique_ptr<>.

native/cpp/CommonCpp/NativeModules/MessageOperationsUtilities/MessageOperationsUtilities.cpp
12 ↗(On Diff #12486)

According to doc: https://github.com/facebook/folly/blob/main/folly/docs/Dynamic.md, to use find() we would need to check ti against .items().end():
json.find("key") != json.items().end() not just json.find("key") != json.end() as we would do with std::map. .count() method returns 0 if and only if there is no object under the given key and it shorter to use in if statements. That is why I preferred to use count()

native/cpp/CommonCpp/NativeModules/MessageOperationsUtilities/MessageOperationsUtilities.cpp
12 ↗(On Diff #12486)

Additionally we use also a regular find() != end() on std::map in this function so it might be confusing for someone to see find() being checked against end() and items().end() somewhere else.

Use if instead of ternary, move only unique_ptrs before return.

tomek added inline comments.
native/cpp/CommonCpp/NativeModules/MessageOperationsUtilities/MessageOperationsUtilities.cpp
12 ↗(On Diff #12486)

Ok, that makes sense. There might be some performance differences, but I don't think they could be significant in our use-case.

17–20 ↗(On Diff #12552)

I think it will be more concise to use reset. What do you think about it?

38 ↗(On Diff #12552)

I don't think adding infos to the name makes sense. Instead it would be great if we can make media plural... but the problem is that media is already plural. Using medias sounds strange, so maybe mediaVector is a good idea?

41 ↗(On Diff #12552)

Maybe just media?

This revision is now accepted and ready to land.May 13 2022, 5:53 AM
native/cpp/CommonCpp/NativeModules/MessageOperationsUtilities/MessageOperationsUtilities.cpp
17–20 ↗(On Diff #12552)

Actually we cant use reset this way since reset requires a pointer as an argument. We could use reset here in one of three ways:
localID.reset((std::make_unique<std::string>(rawMessageInfo["localID"].asString())).get());
or
localID.reset(&(std::string() = rawMessageInfo["localID"].asString()));
or declare a string variable above and use its address as an argument:
std::string localIDvalue = rawMessageInfo["localID"].asString();
localID.reset(&localIDValue);
The second version is shortest but the third one is probably the most readable. I would opt for one of these two.

Rebase and move to persistent storage utilities