Enable serverless in Delta Live Tables in Azure Databricks?

[jesonora]

I'm trying to create a serverless DLT, as far I understood is in Public Review as listed here, Azure Databricks regions - Azure Databricks | Microsoft Learn. I've created a workspace in North Europe, but I cannot see the feature in preview.

Could you help me?

[Kaniz_Fatma]

Hi @jesonora, It appears you’re interested in serverless Delta Live Tables (DLT) in Azure Databricks.

Let me guide you through the process.

1. Serverless DLT Pipelines:

   • Serverless DLT pipelines are currently in Public Preview. They allow you to run your Delta Live Tables pipelines without configuring and deploying infrastructure.
   • With serverless DLT pipelines, you can focus on implementing data ingestion and transformation, while Azure Databricks efficiently manages compute resources.
   • Key capabilities include:
     • Automatically optimized compute: It runs only when needed.
     • Reliable and fully managed compute resources.
     • Efficient dataset updates with incremental refresh for materialized views.
     • Faster startup for compute resources during pipeline updates.
   • Additionally, serverless DLT pipelines offer features like stream pipelining (improving utilization for streaming data workloads) and vertical autoscaling (allocating cost-efficient instance types).
   • To enable serverless DLT pipelines, ensure your workspace has Unity Catalog enabled and is in a serv...¹.

2. Tutorial:

   • You can configure a Delta Live Tables pipeline from code in a Databricks notebook and run it by triggering a pipeline update. Check out this tutorial for step-by-step instructions.

3. Enabling Serverless Mode:

   • In Databricks, to enable serverless pipelines:
     • Click Delta Live Tables in the sidebar.
     • Click Create Pipeline.
     • Give the pipeline a name.
     • Optionally, select the Serverless checkbox to use fully managed compute for this pipeline².

Feel free to explore these resources, and let me know if you have any further questions! 😊

[jesonora]

Hi @Kaniz_Fatma,

Thanks for the quick response. I have checked my region is on Public Preview for DLT, North Europe. But I am not able to see the "Serverless" checkbox.

Am I missing some detail?

Thanks!

[Kaniz_Fatma]

Hi @jesonora, 

• Confirm that Unity Catalog is enabled for your workspace. You can check this by navigating to your Azure Databricks workspace settings.
• If it’s not enabled, you’ll need to enable it to access serverless DLT pipelines.
• You mentioned that your workspace is in North Europe, which is a serverless-enabled region. That’s great!
• However, sometimes changes take a little time to propagate. Please ensure that you’ve waited a reasonable amount of time after creating your workspace in North Europe.
• Verify that you have the necessary permissions to create serverless DLT pipelines. You should have at least Contributor or Owner access to the Azure Databricks workspace.

Let me know how it goes, and feel free to share any additional details. 🚀🔍