Overview
Stream Processing refers to the real-time processing of “data in motion”, that is, performing computations on data as it is being received. Such data is read as continuous streams from data sources such as sensor events, website user activity, financial trades, credit card swipes, click streams, etc. Stream Processing frameworks are able to read large volumes of incoming data and provide valuable insights almost instantaneously.
In this instructor-led, live training (onsite or remote), participants will learn how to set up and integrate different Stream Processing frameworks with existing big data storage systems and related software applications and microservices.
By the end of this training, participants will be able to:
- Install and configure different Stream Processing frameworks, such as Spark Streaming and Kafka Streaming.
- Understand and select the most appropriate framework for the job.
- Process of data continuously, concurrently, and in a record-by-record fashion.
- Integrate Stream Processing solutions with existing databases, data warehouses, data lakes, etc.
- Integrate the most appropriate stream processing library with enterprise applications and microservices.
Audience
- Developers
- Software architects
Format of the Course
- Part lecture, part discussion, exercises and heavy hands-on practice
Notes
- To request a customized training for this course, please contact us to arrange.
Requirements
- Programming experience in any language
- An understanding of Big Data concepts (Hadoop, etc.)
Course Outline
Introduction
- Stream processing vs batch processing
- Analytics-focused stream processing
Overview Frameworks and Programming Languages
- Spark Streaming (Scala)
- Kafka Streaming (Java)
- Flink
- Storm
- Comparison of Features and Strengths of Each Framework
Overview of Data Sources
- Live data as a series of events over time
- Historical data sources
Deployment Options
- In the cloud (AWS, etc.)
- On premise (private cloud, etc.)
Getting Started
- Setting up the Development Environment
- Installing and Configuring
- Assessing Your Data Analysis Needs
Operating a Streaming Framework
- Integrating the Streaming Framework with Big Data Tools
- Event Stream Processing (ESP) vs Complex Event Processing (CEP)
- Transforming the Input Data
- Inspecting the Output Data
- Integrating the Stream Processing Framework with Existing Applications and Microservices
Troubleshooting
Summary and Conclusion