Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Hi @jesonora, It appears youโre interested in serverless Delta Live Tables (DLT) in Azure Databricks.
Let me guide you through the process.
Serverless DLT Pipelines:
Serverless DLT pipelines are currently in Public Preview. They allow you to run your Delta Live Tables pipelines without configuring and deploying infrastructure.
With serverless DLT pipelines, you can focus on implementing data ingestion and transformation, while Azure Databricks efficiently manages compute resources.
Key capabilities include:
Automatically optimized compute: It runs only when needed.
Reliable and fully managed compute resources.
Efficient dataset updates with incremental refresh for materialized views.
Faster startup for compute resources during pipeline updates.
Additionally, serverless DLT pipelines offer features like stream pipelining (improving utilization for streaming data workloads) and vertical autoscaling (allocating cost-efficient instance types).
You can configure a Delta Live Tables pipeline from code in a Databricks notebook and run it by triggering a pipeline update. Check out this tutorial for step-by-step instructions.
Thanks for the quick response. I have checked my region is on Public Preview for DLT, North Europe. But I am not able to see the "Serverless" checkbox.
Confirm thatUnity Catalogis enabled for your workspace. You can check this by navigating to your Azure Databricks workspace settings.
If itโs not enabled, youโll need to enable it to access serverless DLT pipelines.
You mentioned that your workspace is in North Europe, which is a serverless-enabled region. Thatโs great!
However, sometimes changes take a little time to propagate. Please ensure that youโve waited a reasonable amount of time after creating your workspace in North Europe.
Verify that you have the necessary permissions to create serverless DLT pipelines. You should have at leastContributororOwneraccess to the Azure Databricks workspace.
Let me know how it goes, and feel free to share any additional details. ๐๐
Connect with Databricks Users in Your Area
Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโt want to miss the chance to attend and share knowledge.
If there isnโt a group near you, start one and help create a community that brings people together.