[reports-service] Increase RAM and JSON payload size
ClosedPublic
Actions

Authored by bartek on Nov 8 2023, 6:03 AM.

Details

Reviewers

ashoat
michal

Commits

rCOMMeb1a7aa087b0: [reports-service] Increase RAM and JSON payload size

Summary

Attempt to fix ENG-5682. Increased reports service payload limit to be at least MAX_REPORT_LENGTH from lib/utils/report-utils.js. Also increased service RAM to 512MB to avoid potential OOM errors.

Test Plan

None, I'll deploy this and see if it fixes the issue.

Diff Detail

Repository

rCOMM Comm

Lint

Lint Not Applicable

Unit

Tests Not Applicable

Event Timeline

bartek created this revision.Nov 8 2023, 6:03 AM

Herald added subscribers: will, tomek. · View Herald TranscriptNov 8 2023, 6:03 AM

Harbormaster completed remote builds in B23902: Diff 32939.Nov 8 2023, 11:24 AM

bartek requested review of this revision.Nov 8 2023, 11:24 AM

Change makes sense, but from our chat it seems like the test plan failed

This revision is now accepted and ready to land.Nov 8 2023, 11:51 AM

Applied short-term solution discussed in ENG-5682,

Moved reports service from EC2 to Fargate backend - for easier memory scaling (we don't have to care about instance memory)
Increased service memory limit
Made max report size configurable by env var, to avoid rebuilding docker image each time

Test plan: deployed this to prod, my 130MB report was successfully processed.

Harbormaster completed remote builds in B24026: Diff 33109.Nov 13 2023, 12:55 AM

Rebase

Harbormaster completed remote builds in B24682: Diff 34099.Dec 1 2023, 4:20 AM

michal accepted this revision.Dec 1 2023, 6:10 AM

Closed by commit rCOMMeb1a7aa087b0: [reports-service] Increase RAM and JSON payload size (authored by bartek). · Explain WhyDec 1 2023, 6:13 AM

This revision was automatically updated to reflect the committed changes.

bartek added a commit: rCOMMeb1a7aa087b0: [reports-service] Increase RAM and JSON payload size.