- 5748 Views
- 2 replies
- 1 kudos
How can I fix this error. ExecutionError: An error occurred while calling o392.mount: java.lang.NullPointerException
Hello, I'm trying to mount my Databricks to my Azure gen 2 data lake to read in data from the container, but I get an error when executing this line of code: dbutils.fs.mount( source = "abfss://resumes@choisysresume.dfs.core.windows.net/", mount_poin...
- 5748 Views
- 2 replies
- 1 kudos
- 1 kudos
checked it with my mount script and that is exactly the same except that I do not put a '/' after dfs.core.windows.netYou might wanna try that.Also, is Unity enabled? Because Unity does not allow mounts.
- 1 kudos
- 4942 Views
- 4 replies
- 0 kudos
Unable to save file in DBFS
I have took the azure datasets that are available for practice. I got the 10 days data from that dataset and now I want to save this data into DBFS in csv format. I have facing an error :" No such file or directory: 'No such file or directory: '/dbfs...
- 4942 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi,after some exercise you need to aware folder create in dbutils.fs.mkdirs("/dbfs/tmp/myfolder") it's created in /dbfs/dbfs/tmp/myfolderif you want to access path to_csv("/dbfs/tmp/myfolder/mytest.csv") you should created with this script dbutils.fs...
- 0 kudos
- 8602 Views
- 5 replies
- 2 kudos
Resolved! MergeSchema Not Working
Hi All, I have a scenario where my Exisiting Delta Table looks like below:Now I have an incremental data with an additional column i.e. owner:Dataframe Name --> scdDFBelow is the code snippet to merge Incremental Dataframe to targetTable, but the new...
- 8602 Views
- 5 replies
- 2 kudos
- 2 kudos
@Vidula Khanna​ Enabling the below property resolved my issue:spark.conf.set("spark.databricks.delta.schema.autoMerge.enabled",True) Thanks v much!
- 2 kudos
- 8271 Views
- 5 replies
- 6 kudos
Resolved! on-behalf-of token creation (for SPN)
I am trying to create an on-behalf-token for and SPN on my Azure Databricks Premium instance. The response is a FEATURE_DISABLED error message ("On-behalf-of token creation for service principals is not enabled for this workspace"). How do I turn on ...
- 8271 Views
- 5 replies
- 6 kudos
- 6 kudos
There is no On-behalf-of token on Azure - just generate an AAD token for the Service Principal and use it to create PAT (make sure that SP has permission to use PATs). The easiest way of doing it is to use the new Databricks CLI that supports unified...
- 6 kudos
- 2170 Views
- 2 replies
- 1 kudos
Resolved! Job ID value out of range - Azure Logic App Connector
Hello everybody,i tried to build a Logic App Custom Connector following this one explanation. (https://medium.com/@poojaanilshinde/create-azure-logic-apps-custom-connector-for-azure-databricks-e51f4524ab27)Now i run in the following Problem and wante...
- 2170 Views
- 2 replies
- 1 kudos
- 1 kudos
Hey Lukas,I can totally relate to the frustration of encountering those confounding errors when building custom connectors in Azure Logic Apps. The "Job ID value out of range" issue can be quite perplexing, but fear not, for there's a solution on the...
- 1 kudos
- 5876 Views
- 8 replies
- 17 kudos
Spark Structured Streaming : Data write is too slow into adls.
I'm a bit new to spark structured streaming stuff so do ask all the relevant questions if I missed any.I have a notebook which consumes the events from a kafka topic and writes those records into adls. The topic is json serialized so I'm just writing...
- 5876 Views
- 8 replies
- 17 kudos
- 17 kudos
Hi @UmaMahesh1 , • Spark Structured Streaming interacts with Kafka in a certain way, leading to the observed behaviour.• The parameter maxOffsetsPerTrigger in Spark Structured Streaming determines the maximum rate of data read from Kafka.• However, ...
- 17 kudos
- 2922 Views
- 3 replies
- 0 kudos
Resolved! Azure Databricks - Difference between protecting the WEB UI with IP Access list or disabling public access?
Hi, Thoroughly investigating the best security practices for accessing the Databricks WEB UI. I have doubts about the difference between protecting the WEB UI with (1) IP Access list (https://learn.microsoft.com/en-us/azure/databricks/security/networ...
- 2922 Views
- 3 replies
- 0 kudos
- 0 kudos
"In short, would it be the same to configure only the IP of the private endpoint in the IP access list vs disable public access?"The access list doesn't apply to private IPs, only to public IP (internet). Relevant part from the docs:"If you use Priva...
- 0 kudos
- 9741 Views
- 4 replies
- 8 kudos
Resolved! Creating a service principal with admin role on account level in Azure Databricks using Terraform
Dear Community,In the GUI I can grant the admin role to a service principal with a simple switch.How can I achive the same in Terraform? Do you have some code examples?
- 9741 Views
- 4 replies
- 8 kudos
- 8 kudos
Hi @Adrian Wyss​​, It would mean a lot if you could select the "Best Answer" to help others find the correct answer faster.This makes that answer appear right after the question, so it's easier to find within a thread.It also helps us mark the questi...
- 8 kudos
- 3008 Views
- 4 replies
- 3 kudos
Resolved! Error on Azure-Databricks write RDD to storage account with wsabs://
Hi, I'm trying to write data from RDD to the storage account:Adding storage account key:spark.conf.set("fs.azure.account.key.y.blob.core.windows.net", "myStorageAccountKey")Read and write to the same storage:val path = "wasbs://x@y.blob.core.windows....
- 3008 Views
- 4 replies
- 3 kudos
- 3 kudos
Hello @Vadim1 and @User16764241763. I'm wondering if you find a way to avoid adding the hardcoded key in the advanced options spark config section in the cluster configuration. Is there a similar command to spark.conf.set("spark.hadoop.fs.azure.accou...
- 3 kudos
- 5103 Views
- 6 replies
- 1 kudos
Resolved! On-Premise SQL Server Ingestion to Databricks Bronze Layer
Hello everyone!So I want to ingest tables with schemas from the on-premise SQL server to Databricks Bronze layer with Delta Live Table and I want to do it using Azure Data Factory and I want the load to be a Snapshot batch load, not an incremental lo...
- 5103 Views
- 6 replies
- 1 kudos
- 1 kudos
Hi @Parsa Bahraminejad​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best an...
- 1 kudos
- 3237 Views
- 3 replies
- 0 kudos
How to set the permissions to databricks jobs that created and run from Azure Data Factory(ADF)?
I would like to set the permissions to jobs such as granting "CAN_VIEW" or "CAN_MANAGE" to specific groups that run from ADF. It appears that we need to set permissions in pipe line where job runs from ADF, But I could not figure it out. ​​
- 3237 Views
- 3 replies
- 0 kudos
- 0 kudos
Thank you @Debayan Mukherjee​ and @Vidula Khanna​ for getting back to me. But, it didn't help my case. I am specifically looking for setting permissions to the job so that our team can see the job cluster including Spark UI with that privilege. ...
- 0 kudos
- 4167 Views
- 6 replies
- 3 kudos
Azure Storage Account inside Databricks cannot enable soft-delete.
Hello,When deploying any databricks workspace inside Azure, the storage account inside the databricks managed resource group is unable to apply any changes, including enabling soft-delete. Is there a way to enable it?Best regards,Jon
- 4167 Views
- 6 replies
- 3 kudos
- 1196 Views
- 1 replies
- 0 kudos
CDM connector for Spark can't connect to Azure storage account
Hello,I'm trying to use the CDM connector for Spark, but I can't connecto to the Azure storage account when using the connector. I mounted a container of storage account with a SAS-token. When I'm trying to read CDM data from a (mounted) storage acco...
- 1196 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Martijn de Bruijn​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 0 kudos
- 4881 Views
- 2 replies
- 1 kudos
Resolved! Constructor public org.apache.spark.SparkConf(boolean) is not whitelisted.
My code:from great_expectations.datasource import SparkDFDatasourcefrom pyspark.sql import SparkSessionsession_name = 'mk_spark_session'spark = SparkSession.builder.appName(session_name).getOrCreate()datasource = SparkDFDatasource(spark)query = "SELE...
- 4881 Views
- 2 replies
- 1 kudos
- 1 kudos
Are you using a high concurrency cluster? if you do, please try to run this code on standard cluster
- 1 kudos
- 14709 Views
- 5 replies
- 8 kudos
How Selenium Webdriver works on Azure Databricks? I am unable to run a simple code.
from selenium import webdriverfrom webdriver_manager.chrome import ChromeDriverManagerfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.chrome.options import Optionsdrivers = webdriver.Chrome(ChromeDriverManager().install())drivers.g...
- 14709 Views
- 5 replies
- 8 kudos
- 8 kudos
I also got that error. What worked for me was downloading the chrome driver and ensuring its the latest version with shell scripts in the same notebook I used for web scraping. I could not use the web driver manager. You can see all the details here...
- 8 kudos