This introduces the cloudwatch alarms for each identity metric filter.
Depends on D11853
Details
Details
- Reviewers
varun bartek - Commits
- rCOMM62b0317c350e: [terraform] identity service cloudwatch alarms
Tested on staging by triggering identity Search Error error log and activated the cloudwatch alarm.
As part of my final testing task, will test each log pattern individually before landing
Diff Detail
Diff Detail
- Repository
- rCOMM Comm
- Lint
Lint Not Applicable - Unit
Tests Not Applicable
Event Timeline
| services/terraform/remote/aws_cloudwatch_alarms.tf | ||
|---|---|---|
| 6 ↗ | (On Diff #39727) | Isn't this threshold too low? On Staging, I get spammed even when the threshold is 5, 8, and sometimes even 20. |
| services/terraform/remote/aws_cloudwatch_alarms.tf | ||
|---|---|---|
| 6 ↗ | (On Diff #39727) | Me and Varun discussed this and plan on having the threshold set low at first and configure certain thresholds when they come up on staging. An idea I had for this is to actually put the threshold in identity_error_patterns, allowing us to easily configure thresholds for certain error types. I think you bring up an important point on procedure for testing on staging. Perhaps similar to our terraform channel, we'd have devs notify everyone to ignore certain error types they might trigger on purpose? |
| 9 ↗ | (On Diff #39727) | It looks like we'll be using a separate errorType field in our logs. I believe this means the pattern can just be something like Search Error without : or any regex |
| services/terraform/remote/aws_cloudwatch_alarms.tf | ||
|---|---|---|
| 9 ↗ | (On Diff #39727) | Besides for the DB Error error types which will include a * at the start as there's multiple kinds |