Now that the Kinesis data stream is scaling, we’ll take a look at how we can identify this overloading situation by looking at the metrics of the stream. To this end, we’ll create a CloudWatch dashboard that displays some of the relevant metrics.
Navigate to the CloudWatch console, select Dashboards in the navigation pane on the left, and click on Create dashboard
In the create dashboard dialog, enter
beam-workshop as Dashboard name and confirm with Create dashboard
Choose the Line widget
Choose “metrics” and confirm with Configure
beam-workshop in the search field and select Kinesis > Stream metrics
Select the IncomingRecords and GetRecords.IteratorAgeMilliseconds metrics and navigate to Graphed metrics
Adjust the period of both metrics to 1 Minute and change the Statistic to Sum and Maximum, respectively. Switch the Y Axis of the GetRecords.IteratorAgeMilliseconds metric and confirm with Create.
The dashboard now shows two metrics: the amount of incoming messages and the millisBehindLatest metric. The metric reports the time difference between the oldest record currently read by the Kinesis Data Analytics application and the latest record in the stream according to the ingestion time in milliseconds. So it indicates how much behind the processing is from the tip of the stream.
Other important metrics include,
WriteProvisionedThroughputExceeded. You can add them to the dashboard as well to see how the producer is throttled.