Advanced Scaling Components

Application Autoscaling CW Alarm

The CloudFormation Template includes the following components:

  1. Two CloudWatch Alarms for Scale Out and Scale In of your Kinesis Data Analytics for Apache Flink Application.
  2. Accompanying Scale Out and Scale In Autoscaling policy actions in these alarms.
  3. API Gateway Endpoint for triggering AWS Lambda (scale in and scale out)
  4. Lambda function responsible for handling scale in and scale out functions.

Each of these components work in tandem to monitor the metric configured in the CloudWatch Alarm and respond to metrics accordingly. This existing alarm is configured to scale out and in based on the IncomingRecords on the existing Kinesis Data Stream.

Let’s see it in action by sending some data to the stream to observe behavior.

But first, let’s modify the CloudWatch Scale Out alarm to be more sensitive to incoming records.

Currently, the settings are tuned to the max throughput per KPU, which is ideal for a production workload. For this lab, let’s tune this setting down to a lower value to more quickly see results.

1. Navigate to the CloudWatch Service Page

2. Modify CloudWatch Alarm

3. Ingest Data to Kinesis Data Stream and Observe Autoscaling

Once you’ve validated that it has scaled out, please stop the producer on your remote desktop, otherwise your application will continue to scale.

Turning off the producer should show an inverse effect, causing the application to trigger the KDAScaleInAlarm, and the application parallelism should scale back down to 1 within 2 minutes!