Data engineering on Databricks comes together under Lakeflow. Bring data in from almost anywhere with Lakeflow Connect, turn workloads into reliable production jobs with Lakeflow Jobs, and build ETL pipelines visually with Lakeflow Designer, all on the Databricks Data Intelligence Platform.
The catalog covers three Lakeflow courses on the Data Engineer track, two of them with a free self-paced option. Follow the flow below. 👇
Data Engineer · Associate
Data Ingestion with Lakeflow Connect
A comprehensive introduction to Lakeflow Connect for ingesting data into Databricks at scale. Work with Standard and Managed connectors and with batch, incremental batch, and streaming ingestion. Load from cloud object storage using CTAS, COPY INTO, and Auto Loader, handle mismatched records with the rescued data column, flatten semi-structured JSON, and bring in data from databases and SaaS apps. Rounds out with Partner Connect, MERGE INTO, and the Databricks Marketplace, all on UC tables and the medallion architecture.
Data Engineer · Associate
Deploy Workloads with Lakeflow Jobs
Orchestrate and automate data, analytics, and AI workflows with Lakeflow Jobs as a unified orchestration platform. Design workloads as Directed Acyclic Graphs (DAGs), configure scheduling, and use advanced features such as conditional task execution, run-if dependencies, and for-each loops. Covers best practices for production-ready pipelines: compute selection, modular orchestration, error handling, and fault-tolerant design.
Instructor-led available in English, 日本語, Português BR, 한국어, Español, and française.
Go further: build a full medallion pipeline on a visual canvas, with an AI assistant writing the transformations for you. Paid / subscription lab.
Data Engineer · Professional
No-Code ETL with Lakeflow Designer
Build a complete medallion ETL workflow, bronze to silver to gold, visually and without writing pipeline code. Assemble pipelines on a visual canvas using operators, and use the built-in AI authoring assistant, Genie Code, to generate operators from natural language. Clean and reshape data into silver, aggregate and join into gold, schedule the workflow, then deliver insights through an AI/BI dashboard, a Genie Space, and Genie One. An optional bonus uses the AI Function operator to derive sentiment from free-form feedback.
Ingest. Orchestrate. Build. One data engineering experience on the Databricks Data Intelligence Platform.
