I tried my hand at using remote derive to implement the Deserialization trait. However, many of the structs are non-exhaustive, preventing usage of constructors outside of the crate. This means I'm unable to implement structs with the remote attribute. Tried looking but I don't know if there's any solutions to this.

Instead, I just opted to writing my own structs.

• Replaced all unwraps with expects
• Removed all prior leftover comments
• Utilized dedicated types for deserialization of payload
• Replaced usage of log with tracing
• Moved constants to their own separate file

what happens if we hit an expect and panic? wondering if we need to handle errors more gracefully

Since lambda is "serverless", I assume it's one of two scenarios:

• This single func call fails, unaffecting other function calls that are running at the same time. That's okay
• This kills the whole function "instance" (kind of VM for func calls) so other simultaneous calls on this instance gets terminated too. This is bad.

I have a feeling that introducing custom error types is kinda overkill here, the codebase is rather small. Do you think that anyhow context would fit better here? Then we could map anyhow::Error to lambda_runtime::Error in main(). In fact this is just Box<dyn Error>.

In D9936#294586, @bartek wrote:

what happens if we hit an expect and panic? wondering if we need to handle errors more gracefully

Since lambda is "serverless", I assume it's one of two scenarios:

• This single func call fails, unaffecting other function calls that are running at the same time. That's okay
• This kills the whole function "instance" (kind of VM for func calls) so other simultaneous calls on this instance gets terminated too. This is bad.

I have a feeling that introducing custom error types is kinda overkill here, the codebase is rather small. Do you think that anyhow context would fit better here? Then we could map anyhow::Error to lambda_runtime::Error in main(). In fact this is just Box<dyn Error>.

A single error from the lambda can block further processing of other entries for up to a day by default.

We can configure this behavior in our terraform configuration. There are a few options:

1. Setting smaller batches sizes so that errors get isolated so we can quickly move onto new batches
2. Setting retry limits / maximum age limit

I think we could definitely use anyhow

In D9936#300551, @varun wrote:

We can configure this behavior in our terraform configuration. There are a few options:
Setting smaller batches sizes so that errors get isolated so we can quickly move onto new batches
Setting retry limits / maximum age limit

is there a task tracking this?

When we use anyhow and Result instead of panicking, this doesn't matter. Returning Err from main() doesn't kill the whole instance (batch?), just fails a single invocation.

In D9936#300551, @varun wrote:

Not sure why we've converted expect()'s to unwrap()'s.

We can configure this behavior in our terraform configuration. There are a few options:
Setting smaller batches sizes so that errors get isolated so we can quickly move onto new batches
Setting retry limits / maximum age limit

is there a task tracking this?

I've addressed part of this in [4/n] with:

resource "aws_lambda_function_event_invoke_config" "example" {
  function_name                = aws_lambda_function.search_index_lambda.function_name
  maximum_event_age_in_seconds = 60
  maximum_retry_attempts       = 2
}

When running prior batches, I was encountering problems with it running with too many retry attempts. Here's an issue I modified with the current context:
https://linear.app/comm/issue/ENG-5491/setup-error-handling-with-aws-lambda-function