Serverless Data Processing with Dataflow Training
  1. Home
  2. Courses
  3. Google Cloud Platform (GCP)
  4. Serverless Data Processing with Dataflow

Serverless Data Processing with Dataflow

Have a query?

+91 9650575554

Learn to process and transform data at scale with Serverless Data Processing using Google Cloud Dataflow. Master Apache Beam and build cost-effective, real-time data pipelines.

Let's Lalk

Overview

The Serverless Data Processing with Dataflow training enables you to design and deploy scalable data pipelines using Google Cloud Dataflow. This course covers key topics like Apache Beam programming, stream and batch data processing, pipeline optimization, and integration with other Google Cloud services. With hands-on labs, real-world use cases, and expert guidance, you'll gain practical experience to efficiently process large datasets in real-time and batch modes. Ideal for data engineers and developers looking to simplify and scale their data processing tasks on Google Cloud.

Key Features

  • Expert Instruction
  • Integration Skills
  • Optimization Techniques
  • Stream and Batch Processing

Curriculum

Study Plan

1Module 1: Introduction
  • Course Introduction 
  • Beam and Dataflow Refresher 
2Module 2: Beam Portability
  • Beam Portability
  • Runner v2 
  • Container Environments
  • Cross-Language TransformS  
3Module 3: Separating Compute and Storage with Dataflow
  • Dataflow 
  • Dataflow Shuffle Service
  • Dataflow Streaming Engine
  • Flexible Resource Scheduling 
4Module 4: IAM, Quotas, and Permissions
  • IAM 
  • Quota 
5Module 5: Security
  • Data Locality 
  • Shared VPC 
  • Private IPs 
  • CMEK
6Module 6: Beam Concepts Review
  • Beam Basics
  • Utility Transforms 
  • DoFn Lifecycle  
7Module 7: Module 7 Windows, Watermarks, Triggers
  • Windows 
  • Watermarks 
  • Triggers 
8Module 8: Sources and Sinks
  • Sources and Sinks
  • Text IO and File IO
  • BigQuery IO
  • PubSub IO
  • Kafka IO 
  • Bigable IO 
  • Avro IO
  • Splittable DoFn 
9Module 9: Schemas
  • Beam Schemas
  • Code Example
10Module 10: State and Timers
  • State API
  • Timer API 
  • Summary
11Module 11: Best Practices
  • Schemas
  • Handling unprocessable Data
  • Error Handling 
  • AutoValue Code Generator
  • JSON Data Handling 
  • Utilize DoFn Lifecycle 
  • Pipeline Optimizations
12Module 12: Dataflow SQL and DataFrames
  • Dataflow and Beam SQL 
  • Windowing in SQL 
  • Beam DataFrames
13Module 13: Beam Notebooks
  • Beam Notebooks
14Module 14 Beam Notebooks
  • Job List
  • Job Info
  • Job Graph 
  • Job Metrics
  • Metrics Explorer 
15Module 15: Logging and Error Reporting
  • Logging
  • Error Reporting 
16Module 16: Troubleshooting and Debug
  • Troubleshooting Workflow 
  • Types of Troubles
17Module 17: Performance
  • Pipeline Design 
  • Data Shape
  • Source, Sinks, and External Systems
  • Shuffle and Streaming Engine 
18Module 18 Testing and CI/CD
  • Testing and CI/CD Overview
  • Unit Testing 
  • Integration Testing
  • Artifact Building 
  • Deployment 
19Module 19: Reliability
  • Introduction to Reliability 
  • Monitoring 
  • Geolocation 
  • Disaster Recovery
  • High Availability
20Module 20: Flex Templates
  • Classic Templates
  • Flex Templates 
  • Using Flex Templates
  • Google-provided Templates 

About Instructor

Still have question

Still have question?

Top instructors from around the world teach millions of students on CloudFreeks.

Let's talk Whatsapp
Become An Instructor Today

Become An Instructor Today

Top instructors from around the world teach millions of students on CloudFreeks.

Start Today