- 9867 Views
- 3 replies
- 1 kudos
How can I fix this error. ExecutionError: An error occurred while calling o392.mount: java.lang.NullPointerException
Hello, I'm trying to mount my Databricks to my Azure gen 2 data lake to read in data from the container, but I get an error when executing this line of code: dbutils.fs.mount( source = "abfss://resumes@choisysresume.dfs.core.windows.net/", mount_poin...
- 9867 Views
- 3 replies
- 1 kudos
- 1 kudos
I was using databricks scopes, to get the key which was used in the the config. I received a similar mount error while mounting with "wasbs" driver, "ExecutionError: An error occurred while calling o427.mount.", this was the issue because the scope ...
- 1 kudos
- 14864 Views
- 5 replies
- 0 kudos
Help With OSError: [Errno 95] Operation not supported: '/Workspace/Repos/Connectors....
Hello,I am experiencing issues with importing from utils repo the schema file I created.this is the logic we use for all ingestion and all other schemas live in this repo utills/schemasI am unable to access the file I created for a new ingestion pipe...
- 14864 Views
- 5 replies
- 0 kudos
- 0 kudos
@ArbMake sure the below is a file with extension .py and not a notebook. That should resolve the issue./Workspace/Repos/Connectors/Dev/utils/schemas/Comptroller.py'
- 0 kudos
- 8065 Views
- 6 replies
- 4 kudos
DLT | Cluster terminated by System-User | INTERNAL_ERROR: Communication lost with driver. Cluster 0312-140502-k9monrjc was not reachable for 120 seconds
Dear Community, Hope you are doing well.For the last couple of days I am seeing very strange issues with my DLT pipeline, So every 60-70 mins it is getting failed in continuous mode, with the ERROR; INTERNAL_ERROR: Communication lost with driver. Clu...
- 8065 Views
- 6 replies
- 4 kudos
- 4 kudos
We had similar error for one the DLT pipeline, This could be some times because of compute size, we had increased compute size of server in your DLT pipelines, still we were seeing this error while processing very large file. we then added below para...
- 4 kudos
- 7579 Views
- 2 replies
- 1 kudos
Are there Visio stencils for Databricks?
I need to create some Visio's to show different processes in Databricks. Microsoft offers a Azure pack but it doesn't include Databricks.
- 7579 Views
- 2 replies
- 1 kudos
- 1 kudos
@Christopher Shehu - I am so sorry. I don't know how your question slid under my radar. Hang in there. I'll try to find someone to answer your question as quickly as possible. Thank you for your patience!
- 1 kudos
- 8171 Views
- 6 replies
- 5 kudos
Resolved! Unity Catalog - Azure account console - how to access?
I'm trying to access account console in Azure but I only can see the list of workspaces and access them. I didn't find documentation about account console for Azure. Do you know how to access account console?
- 8171 Views
- 6 replies
- 5 kudos
- 5 kudos
Hello @Leszek . Please tell me is it works for you ?Did you find the root cause ?I still don't understand why I should grant to myself some extra permissions if I already global administrator, owner of subscription, owner of databricks workspace but...
- 5 kudos
- 6039 Views
- 2 replies
- 2 kudos
How to read a PDF file from Azure Datalake blob storage to Databricks
I have a scenario where I need to read a pdf file from "Azure Datalake blob storage to Databricks", where connection is done through AD access.Generating the SAS token has been restricted in our environment due to security issues. The below script ca...
- 6039 Views
- 2 replies
- 2 kudos
- 2 kudos
@PunithRaj You can try to use PDF DataSource for Apache Spark for read pdf files directly to the DataFrame. So you will have extracted text and rendered page as image in output. More details here: https://stabrise.com/spark-pdf/df = spark.read.forma...
- 2 kudos
- 10634 Views
- 4 replies
- 1 kudos
How to read data from Azure Log Analitycs ?
Hi guys,I need to read data from Azure Log Analitycs Workspace directaly, have any idea ?thank you
- 10634 Views
- 4 replies
- 1 kudos
- 1 kudos
You can use Kusto Spark connector for that: https://github.com/Azure/azure-kusto-spark/blob/master/docs/KustoSource.md#source-read-command It heavily depends on how you access data, there could be a need for using ADX cluster for it: https://learn.mi...
- 1 kudos
- 43861 Views
- 26 replies
- 14 kudos
Resolved! How to use Databricks Repos with a service principal for CI/CD in Azure DevOps?
Databricks Repos best-practices recommend using the Repos REST API to update a repo via your git provider. The REST API requires authentication, which can be done one of two ways:A user / personal access tokenA service principal access tokenUsing a u...
- 43861 Views
- 26 replies
- 14 kudos
- 14 kudos
For anyone coming here in the future, this should explain it: https://docs.databricks.com/aws/en/repos/ci-cd-techniques-with-repos#authorize-a-service-principal-to-access-git-foldersBasically:1. Go to your service in settings -> identity and access -...
- 14 kudos
- 4973 Views
- 7 replies
- 7 kudos
Azure DevOps Repo - Invalid Git Credentials
I have a Repo in Databricks connected to Azure DevOps Repositories.The repo has been working fine for almost a month, until last week. Now when I try to open the Git settings in Databricks, I am getting "Invalid Git Credentials". Nothing has change...
- 4973 Views
- 7 replies
- 7 kudos
- 3239 Views
- 5 replies
- 2 kudos
User entry via portal/UI
Hello Experts,We have data bricks on Azure. We need to provide a user interface to users so that some customizing table entries can be entered by end users which in turn are saved in the Delta table. Is there any feature in Databricks or tools that w...
- 3239 Views
- 5 replies
- 2 kudos
- 2 kudos
I achieved this using textbox widget and asked user to enter and run dashboard, in python code validating and inserting data in databricks.2nd method , created excel vba application, using odbc connection to insert delete and update data in delta tab...
- 2 kudos
- 9411 Views
- 7 replies
- 6 kudos
How to connect Cognos 11.1.7 to Azure Databricks
We are trying to connect Cognos 11.1.7 to Azure Databricks, but no success.Can you please help or guide us how to connect Cognos 11.1.7 to Azure Databricks.This is very critical to our user community. Can you please help or guide us how to connect Co...
- 9411 Views
- 7 replies
- 6 kudos
- 6 kudos
Have anyone got the Simba JDBC driver going with CA 11.1.7? The ODBC driver works fine but i can't get the JDBC running.Regd's
- 6 kudos
- 10521 Views
- 5 replies
- 7 kudos
Resolved! Databricks Autoloader is getting stuck and does not pass to the next batch
I have a simple job scheduled every 5 min. Basically it listens to cloudfiles on storage account and writes them into delta table, extremely simple. The code is something like this:df = (spark .readStream .format("cloudFiles") .option('cloudFil...
- 10521 Views
- 5 replies
- 7 kudos
- 7 kudos
I had the same issue: files would randomly not be loaded.Setting `.option("cloudFiles.useIncrementalListing", False)` Seemed to do the trick!
- 7 kudos
- 5566 Views
- 3 replies
- 4 kudos
Resolved! Spot instance in Azure Databricks
When I run a job enabling using spot instances , I would like to know how many number of workers are using spot and how many number of workers are using on demand instances for a given job run In order to identify the spot instances we got for any...
- 5566 Views
- 3 replies
- 4 kudos
- 22796 Views
- 4 replies
- 26 kudos
How to connect your Azure Data Lake Storage to Azure DatabricksStandard Workspace �� Private link In your storage accounts please go to “Networ...
How to connect your Azure Data Lake Storage to Azure DatabricksStandard Workspace Private linkIn your storage accounts please go to “Networking” -> “Private endpoint connections” and click Add Private Endpoint.It is important to add private links in ...
- 22796 Views
- 4 replies
- 26 kudos
- 7367 Views
- 7 replies
- 4 kudos
need help with Azure Databricks questions on CTE and SQL syntax within notebooks
Hi amazing community folks,Feel free to share your experience or knowledge regarding below questions:-1.) Can we pass a CTE sql statement into spark jdbc? i tried to do it i couldn't but i can pass normal sql (Select * from ) and it works. i heard th...
- 7367 Views
- 7 replies
- 4 kudos
- 4 kudos
Not a comparison, but there is a DB-SQL cheatsheet at https://www.databricks.com/sites/default/files/2023-09/databricks-sql-cheatsheet.pdf/
- 4 kudos