Ingest data to Elasticsearch

You can now finally inspect the results that are generated by the Flink application into Elasticsearch and inspect the visualization in Kibana.

  1. Terminate the Java producer application that is still producing data into the Kinesis data stream. Go to the terminal pane in IntelliJ and press CTRL-C until the producer terminates.

  2. Navigate to the main method of the ProcessTaxiStream class and run it. However, as soon as it has started to execute, you can terminate it again by clicking the red square.

  3. Edit the runtime parameters of the main method by choosing Edit Configurations

  4. Under Program arguments, enter --ElasticsearchEndpoint followed by the ElasticsearchEndpoint that you have noted earlier from the Elasticsearch Service console

  5. Confirm with OK and execute the program again by clicking on the green arrow

  6. Restart the Java producer application by executing the following command in the Terminal pane of IntelliJ

    java -jar C:\Users\Administrator\Desktop\workshop-resources\amazon-kinesis-replay-0.1.0.jar -objectPrefix artifacts/kinesis-analytics-taxi-consumer/taxi-trips-partitioned.json.lz4 -aggregate -streamName streaming-analytics-workshop -speedup 600
  7. Navigate to the Dashboard in Kibana and click on nyc-tlc-dashboard to view the visualization of the data generated by the Flink application

    If you cannot see any new data in the visualization, you may need to adapt the time range in the upper left corner of the Kibana dashboard. The output of the Java producer application will tell you the time of events that are currently produced into the Kinesis stream. In addition, make sure to check both the Jar and the Flink application are running.

  8. Once you have verified that the data generated by the Flink application is visualized by Kibana, stop the Flink application in IntelliJ. Also terminate the Java producer application again by navigating to the Terminal in IntelliJ and pressing Ctrl-C.