- 3191 Views
- 2 replies
- 1 kudos
Resolved! Job ID value out of range - Azure Logic App Connector
Hello everybody,i tried to build a Logic App Custom Connector following this one explanation. (https://medium.com/@poojaanilshinde/create-azure-logic-apps-custom-connector-for-azure-databricks-e51f4524ab27)Now i run in the following Problem and wante...
- 3191 Views
- 2 replies
- 1 kudos
- 1 kudos
Hey Lukas,I can totally relate to the frustration of encountering those confounding errors when building custom connectors in Azure Logic Apps. The "Job ID value out of range" issue can be quite perplexing, but fear not, for there's a solution on the...
- 1 kudos
- 8643 Views
- 7 replies
- 17 kudos
Spark Structured Streaming : Data write is too slow into adls.
I'm a bit new to spark structured streaming stuff so do ask all the relevant questions if I missed any.I have a notebook which consumes the events from a kafka topic and writes those records into adls. The topic is json serialized so I'm just writing...
- 8643 Views
- 7 replies
- 17 kudos
- 4383 Views
- 3 replies
- 0 kudos
Resolved! Azure Databricks - Difference between protecting the WEB UI with IP Access list or disabling public access?
Hi, Thoroughly investigating the best security practices for accessing the Databricks WEB UI. I have doubts about the difference between protecting the WEB UI with (1) IP Access list (https://learn.microsoft.com/en-us/azure/databricks/security/networ...
- 4383 Views
- 3 replies
- 0 kudos
- 0 kudos
"In short, would it be the same to configure only the IP of the private endpoint in the IP access list vs disable public access?"The access list doesn't apply to private IPs, only to public IP (internet). Relevant part from the docs:"If you use Priva...
- 0 kudos
- 13281 Views
- 3 replies
- 8 kudos
Resolved! Creating a service principal with admin role on account level in Azure Databricks using Terraform
Dear Community,In the GUI I can grant the admin role to a service principal with a simple switch.How can I achive the same in Terraform? Do you have some code examples?
- 13281 Views
- 3 replies
- 8 kudos
- 8 kudos
Dear @Pat Sienkiewicz , works perfectly! It would be so easy it the documentation would be better... Rg Adrian
- 8 kudos
- 3861 Views
- 3 replies
- 3 kudos
Resolved! Error on Azure-Databricks write RDD to storage account with wsabs://
Hi, I'm trying to write data from RDD to the storage account:Adding storage account key:spark.conf.set("fs.azure.account.key.y.blob.core.windows.net", "myStorageAccountKey")Read and write to the same storage:val path = "wasbs://x@y.blob.core.windows....
- 3861 Views
- 3 replies
- 3 kudos
- 3 kudos
Hello @Vadim1 and @User16764241763. I'm wondering if you find a way to avoid adding the hardcoded key in the advanced options spark config section in the cluster configuration. Is there a similar command to spark.conf.set("spark.hadoop.fs.azure.accou...
- 3 kudos
- 7572 Views
- 6 replies
- 1 kudos
Resolved! On-Premise SQL Server Ingestion to Databricks Bronze Layer
Hello everyone!So I want to ingest tables with schemas from the on-premise SQL server to Databricks Bronze layer with Delta Live Table and I want to do it using Azure Data Factory and I want the load to be a Snapshot batch load, not an incremental lo...
- 7572 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @Parsa Bahraminejad Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best an...
- 1 kudos
- 4796 Views
- 3 replies
- 0 kudos
How to set the permissions to databricks jobs that created and run from Azure Data Factory(ADF)?
I would like to set the permissions to jobs such as granting "CAN_VIEW" or "CAN_MANAGE" to specific groups that run from ADF. It appears that we need to set permissions in pipe line where job runs from ADF, But I could not figure it out.
- 4796 Views
- 3 replies
- 0 kudos
- 0 kudos
Thank you @Debayan Mukherjee and @Vidula Khanna for getting back to me. But, it didn't help my case. I am specifically looking for setting permissions to the job so that our team can see the job cluster including Spark UI with that privilege. ...
- 0 kudos
- 6379 Views
- 6 replies
- 3 kudos
Azure Storage Account inside Databricks cannot enable soft-delete.
Hello,When deploying any databricks workspace inside Azure, the storage account inside the databricks managed resource group is unable to apply any changes, including enabling soft-delete. Is there a way to enable it?Best regards,Jon
- 6379 Views
- 6 replies
- 3 kudos
- 1721 Views
- 1 replies
- 0 kudos
CDM connector for Spark can't connect to Azure storage account
Hello,I'm trying to use the CDM connector for Spark, but I can't connecto to the Azure storage account when using the connector. I mounted a container of storage account with a SAS-token. When I'm trying to read CDM data from a (mounted) storage acco...
- 1721 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Martijn de Bruijn Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 7076 Views
- 2 replies
- 1 kudos
Resolved! Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.
My code:from great_expectations.datasource import SparkDFDatasourcefrom pyspark.sql import SparkSessionsession_name = 'mk_spark_session'spark = SparkSession.builder.appName(session_name).getOrCreate()datasource = SparkDFDatasource(spark)query = "SELE...
- 7076 Views
- 2 replies
- 1 kudos
- 1 kudos
Are you using a high concurrency cluster? if you do, please try to run this code on standard cluster
- 1 kudos
- 16987 Views
- 5 replies
- 8 kudos
How Selenium Webdriver works on Azure Databricks? I am unable to run a simple code.
from selenium import webdriverfrom webdriver_manager.chrome import ChromeDriverManagerfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.chrome.options import Optionsdrivers = webdriver.Chrome(ChromeDriverManager().install())drivers.g...
- 16987 Views
- 5 replies
- 8 kudos
- 8 kudos
I also got that error. What worked for me was downloading the chrome driver and ensuring its the latest version with shell scripts in the same notebook I used for web scraping. I could not use the web driver manager. You can see all the details here...
- 8 kudos
- 7708 Views
- 2 replies
- 1 kudos
Trying to mount Azure Data Lake Storage Gen 2 to Azure Databricks
I have validated many many many times all my credentials and I am still getting the following error (at the very end). ChatGPT said to basically recheck everything and I did. The one thing I didn't do was grant the permissions which I have since done...
- 7708 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Chris Sarrico Could you please specify the container name before the storage account name like this :so the source looks like this. :source = "abfss://<container-name>@<storage-account-name>.dfs.core.windows.net/"https://learn.microsoft.com/en-u...
- 1 kudos
- 6695 Views
- 9 replies
- 14 kudos
Resolved! Pipeline workflow dude
Hi! I have a problem. I'm using an autoloader to ingest data from raw to a Delta Lake, but when my pipeline starts, I want to apply the pipeline only to the new data. The autoloader ingests data into the Delta Lake, but now, how can I distinguish the...
- 6695 Views
- 9 replies
- 14 kudos
- 14 kudos
Hi @Alejandro Piury Pinzón We haven't heard from you since the last response from @Tyler Retzlaff , and I was checking back to see if her suggestions helped you.Or else, If you have any solution, please share it with the community, as it can be he...
- 14 kudos
- 2314 Views
- 3 replies
- 5 kudos
Issue with connection our Alation instance to Azure Databricks using the Unity Catalog OCF connector
When we try and do the above I am able to get the list of schemas. But when I select one to injest we are then getting issue due to it trying to access system.lineage.table_lineage. When I look in the System catalog I can only see a schema called inf...
- 2314 Views
- 3 replies
- 5 kudos
- 5 kudos
We are working with an agency that is a Databricks partner if that is what you mean.
- 5 kudos
- 2150 Views
- 1 replies
- 0 kudos
- 2150 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Sourav Nandy Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos