Introducing Databricks LakeFlow: A unified, intelligent solution for data engineering

Sujitha — Tue, 18 Jun 2024 09:49:54 GMT

Today, we are excited to announce Databricks LakeFlow, a new solution that contains everything you need to build and operate production data pipelines. It includes new native, highly scalable connectors for databases including MySQL, Postgres, SQL Server and Oracle and enterprise applications like Salesforce, Microsoft Dynamics, NetSuite, Workday, ServiceNow and Google Analytics. Users can transform data in batch and streaming using standard SQL and Python. We are also announcing Real Time Mode for Apache Spark, allowing stream processing at orders of magnitude faster latencies than microbatch. Finally, you can orchestrate and monitor workflows and deploy to production using CI/CD. Databricks LakeFlow is native to the Data Intelligence Platform, providing serverless compute and unified governance with Unity Catalog.

LakeFlow is the one unified data engineering solution for ingestion, transformation and orchestration

In this blog post we discuss the reasons why we believe LakeFlow will help data teams meet the growing demand of reliable data and AI as well as LakeFlow’s key capabilities integrated into a single product experience.

Continue to read more.

Re: Introducing Databricks LakeFlow: A unified, intelligent solution for data engineering

Erik — Thu, 20 Jun 2024 15:17:24 GMT

Can you say something more about the "Real Time Mode for Apache Spark"? I can not find anything about it anywhere.

- Is it "just" spark continuous processing?

- Will it be available with "normal" pyspark in databricks, or only through DLT?

- Can it stream to delta tables?

topic Introducing Databricks LakeFlow: A unified, intelligent solution for data engineering in Announcements

Introducing Databricks LakeFlow: A unified, intelligent solution for data engineering

Re: Introducing Databricks LakeFlow: A unified, intelligent solution for data engineering