Monitor important metrics

Now that the Kinesis data stream is scaling, we’ll take a look at how we can identify this overloading situation by looking at the metrics of the stream. To this end, we’ll create a CloudWatch dashboard that displays some of the relevant metrics.

  1. Navigate to the CloudWatch console, click on Dashboards in the navigation pane on the left, and select the BeamWorkshopDashboard dashboard you have been using earlier.

  2. Click on Add widget and choose the Line widget to compare metrics over time. Confirm with Next.

  3. In the next dialog, choose metrics and confirm with Configure

  4. Enter beam-workshop in the search field and select Kinesis > Stream metrics

  5. Select the IncomingRecords and GetRecords.IteratorAgeMilliseconds metrics and navigate to Graphed metrics

  6. Adjust the period of both metrics to 1 Minute and change the Statistic to Sum and Maximum, respectively. Switch the Y Axis of the GetRecords.IteratorAgeMilliseconds metric and confirm with Create.

The dashboard now shows two metrics: the amount of incoming messages and the millisBehindLatest metric. The metric reports the time difference between the oldest record currently read by the Kinesis Data Analytics application and the latest record in the stream according to the ingestion time in milliseconds. So it indicates how much behind the processing is from the tip of the stream.

Other important metrics include, ReadProvisionedThroughputExceeded and WriteProvisionedThroughputExceeded. You can add them to the dashboard as well to see how the producer is throttled.