CodePark/ questions / CloudWatch, logs & X-Ray
START PRACTICINGPRACTICE
Backend question · Cloud & AWS › CloudWatch, logs & X-Ray

Which CloudWatch alarms would you set up for an API on ECS behind an ALB that also processes jobs from SQS, and how should alarms be configured to avoid noise?

Alarm first on the symptoms users feel, then on the causes.

What to alarm on:

  • ALB: the target 5xx count or error rate, the p99 TargetResponseTime (the response time that 99% of requests beat), and UnHealthyHostCount.
  • ECS service: CPU and memory use. These are mostly for tuning scaling, not for waking someone up.
  • SQS main queue: ApproximateAgeOfOldestMessage. A growing age shows the consumers are falling behind.
  • SQS DLQ (dead-letter queue, where failed messages go): ApproximateNumberOfMessagesVisible above zero.
  • RDS: CPU, free storage and connection count.

How to cut noise:

  • Alarm on rates and percentiles, not on single errors.
  • Require several bad datapoints in an evaluation window, such as 3 out of 5 minutes.
  • Decide how the alarm treats missing data.

Send alarms to an SNS topic. The topic pages the on-call engineer or posts to chat. Composite alarms combine several signals, so one incident sends only one page.

  • On the ALB, alarm on the 5xx rate, p99 latency and unhealthy hosts.
  • On SQS, alarm on the age of the oldest message and on a DLQ that is not empty.
  • On RDS, alarm on CPU, free storage and connections.
  • Use percentiles, M-of-N datapoints and missing-data settings to cut noise.
  • Notify through SNS, and use composite alarms to avoid duplicate pages.