A lambda written in Rust that consumes a dynamodb stream for identity-users and indexes users into the identity-search OpenSearch instance.
Depends on D9877
Differential D9936
[services] [3/n] Rust opensearch indexing lambda will on Nov 20 2023, 6:36 PM. Authored by Tags None Referenced Files
Details A lambda written in Rust that consumes a dynamodb stream for identity-users and indexes users into the identity-search OpenSearch instance. Depends on D9877 Successfully built using cargo lambda and deployed with terraform. Confirmed manually the lambda is syncing the stream.
Diff Detail
Event Timeline
Comment Actions I tried my hand at using remote derive to implement the Deserialization trait. However, many of the structs are non-exhaustive, preventing usage of constructors outside of the crate. This means I'm unable to implement structs with the remote attribute. Tried looking but I don't know if there's any solutions to this. Instead, I just opted to writing my own structs. Comment Actions
Comment Actions Thanks for addressing these, good job! I'll let @varun take a look That's unfortunate but yeah, writing own structs is a reasonable solution. You can optionally add a comment that they correspond to DynamoDB Streams types. Comment Actions what happens if we hit an expect and panic? wondering if we need to handle errors more gracefully
Comment Actions
Since lambda is "serverless", I assume it's one of two scenarios:
I have a feeling that introducing custom error types is kinda overkill here, the codebase is rather small. Do you think that anyhow context would fit better here? Then we could map anyhow::Error to lambda_runtime::Error in main(). In fact this is just Box<dyn Error>.
Comment Actions A single error from the lambda can block further processing of other entries for up to a day by default. We can configure this behavior in our terraform configuration. There are a few options:
I think we could definitely use anyhow Comment Actions Not sure why we've converted expect()'s to unwrap()'s.
is there a task tracking this?
Comment Actions When we use anyhow and Result instead of panicking, this doesn't matter. Returning Err from main() doesn't kill the whole instance (batch?), just fails a single invocation. Comment Actions I've addressed part of this in [4/n] with: resource "aws_lambda_function_event_invoke_config" "example" { function_name = aws_lambda_function.search_index_lambda.function_name maximum_event_age_in_seconds = 60 maximum_retry_attempts = 2 } When running prior batches, I was encountering problems with it running with too many retry attempts. Here's an issue I modified with the current context: |