โ03-14-2024 05:08 AM
I have a data bricks workspace and an Azure data lake storage account.
Both are present in the same Vnet.
Unity catalog is enabled in the worksapce.
I have created some tables in unity catalog.
I am able to query the data from the tables when I use the any of the below 2 computes:
1. All purpose cluster
2. SQL Warehouse of type "PRO"
When I try to read the same table data using a SQL warehouse of the type "serverless", I get the below error:
This Azure storage request is not authorized. The storage account's 'Firewalls and virtual networks' settings may be blocking access to storage services. Please verify your Azure storage credentials or firewall exception settings.
I know that the access, firewalls and networking are not a problem because it is working with all purpose cluster and PRO SQL warehouse. So I not know what is the problem here.
Please let me how to fix this.
โ03-14-2024 05:28 AM
but serverless is in Databricks Subscription so you need to whitelist traffic from there.
Please check that list. Storage firewall needs to allow connections https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/serv...
โ03-14-2024 05:28 AM
but serverless is in Databricks Subscription so you need to whitelist traffic from there.
Please check that list. Storage firewall needs to allow connections https://learn.microsoft.com/en-us/azure/databricks/security/network/serverless-network-security/serv...
โ03-15-2024 01:40 AM
Hi @NarenderKumar, Configuring an ETL framework using Delta Live Tables (DLT) can be powerful, especially when you want to maintain flexibility and avoid hardcoding configurations directly in your notebook.
Letโs explore some options for managing your pipeline settings and configurations:
Delta Live Tables UI Configuration:
Product Edition Selection:
Pipeline Source Code:
Parameterization:
Error While Converting to Pandas DataFrame:
Pipeline Execution Sequence:
Iterating Over Rows:
Remember that DLT is designed to simplify ETL workflows, and you have the flexibility to choose the most suitable approach for your use case. Feel free to explore the UI, leverage parameterization, and adapt your pipeline settings to external sources as needed.
If you encounter specific issues, feel free to share more details, and Iโll be happy to assist further! ๐
Excited to expand your horizons with us? Click here to Register and begin your journey to success!
Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!