- 5829 Views
- 3 replies
- 0 kudos
Unable to read excel files in the Azure databricks (UC enabled workspace)
Hello,After adding the maven library com.crealytics:spark-excel_2.12:0.13.5 under the artifact allowlist, I have installed it at the Azure databricks cluster level (shared, unity catalog enabled, runtime 15.4). Then I tried to create a df for the exc...
- 5829 Views
- 3 replies
- 0 kudos
- 0 kudos
I did a little more digging and found further information: Unity Catalog does not natively support reading Excel files directly. Based on the provided context, there are a few key points to consider: Third-Party Libraries: Reading Excel files in D...
- 0 kudos
- 2452 Views
- 3 replies
- 0 kudos
Predictive Optimization is not running
We have enabled predictive optimization at account level and metastore level. The enables check box can be seen in catalog details and table details. When i query the system.storage.predictive_optimization_operations_history table, it is still empty....
- 2452 Views
- 3 replies
- 0 kudos
- 0 kudos
I can't help with your specifc workspace as I don't have access to any customer environment. Support can help if you open a ticket with them but at this point I am out suggestions.
- 0 kudos
- 3533 Views
- 8 replies
- 0 kudos
Not Able To Access GCP storage bucket from Databricks
While running :df = spark.read.format("csv") \ .option("header", "true") \ .option("inferSchema", "true") \ .load('path')df.show()Getting error : java.io.IOException: Invalid PKCS8 data.Cluster Spark Config : spark.hadoop.fs.gs.auth.service....
- 3533 Views
- 8 replies
- 0 kudos
- 0 kudos
At this point it is out of my area of knowledge and I don't havey any further suggestions. You may want to consider contacting Databricks Support if you have a support contract.
- 0 kudos
- 10233 Views
- 2 replies
- 0 kudos
Resolved! Databricks with Private cloud
Hi Databricks Team,Is it possible for Databricks to offer support for private cloud environments other than Azure, GCP, and AWS? The client intends to utilize Databricks in their own cloud for enhanced security. If this is feasible, what is the proce...
- 10233 Views
- 2 replies
- 0 kudos
- 0 kudos
We're looking at Databricks, but would prefer to use a Pure Storage Array to house data, mostly because of the cost of data storage for cloud providers. We're okay using cloud compute, but storage is much more feasible for us with local/private stora...
- 0 kudos
- 5802 Views
- 3 replies
- 2 kudos
Databricks UMF Best Practice
Hi there, I would like to get some feedback on what are the ideal/suggested ways to get UMF data from our Azure cloud into Databricks. For context, UMF can mean either:User Managed FileUser Maintained FileBasically, a UMF could be something like a si...
- 5802 Views
- 3 replies
- 2 kudos
- 2 kudos
I am not an expert on this topic or Azure services but I did some research and have some suggested courses of action for you to test out. To address your request for suggested ways to get User Managed Files (UMF) from Azure into Databricks, here are...
- 2 kudos
- 2738 Views
- 3 replies
- 4 kudos
Resolved! toml file syntax highlighting
Hi there, I'm curious if there's a way for Databricks to support syntax highlighting for a language that is currently not supported in our DBX configuration. For example, I'm using .toml files, but Databricks doesn't understand it and displays it as ...
- 2738 Views
- 3 replies
- 4 kudos
- 4 kudos
Hello @ChristianRRL! Sorry for the delayed response. Databricks currently does not support syntax highlighting for .toml files. As a workaround, you can edit toml files in external editors like VS code (with plugins) and sync them to Databricks using...
- 4 kudos
- 11452 Views
- 3 replies
- 3 kudos
Resolved! Cannot import relative python paths
Hello,Some variations of this question have been asked before but there doesn't seem to be an answer for the following simple use case:I have the following file structure on a Databricks Asset Bundles project: src --dir1 ----file1.py --dir2 ----file2...
- 11452 Views
- 3 replies
- 3 kudos
- 3 kudos
This works as long as the script calling the module is indeed __main__; i've changed it a bit to make it more generic:import os import sys def find_module(path): while path: if os.path.basename(path) == "src": return path ...
- 3 kudos
- 2483 Views
- 3 replies
- 0 kudos
not able to run python script even after everything is in place in azure data bricks
getting the below error while running a python which connects to azure sql db Database connection error: ('01000', "[01000] [unixODBC][Driver Manager]Can't open lib 'ODBC Driver 17 for SQL Server' : file not found (0) (SQLDriverConnect)") can some on...
- 2483 Views
- 3 replies
- 0 kudos
- 0 kudos
The error occurs because the Microsoft ODBC Driver 17 for SQL Server is missing on your Azure Databricks cluster. Here's how to fix it: Steps to Resolve Step 1: Create an Init Script to Install ODBC Driver1. Create a file named `odbc-install.sh` with...
- 0 kudos
- 2206 Views
- 3 replies
- 2 kudos
Resolved! Materialized Views Compute
When creating a Materialized View (MV) without a schedule, there seems to be a cost associated with the MV once it is created, even if it is not queried.The question is, once the MV is created, is there already a "hot" compute ready for use in case a...
- 2206 Views
- 3 replies
- 2 kudos
- 2 kudos
Please select "Accept as Solution" so that others can benefit from this exchange. Regards, Louis.
- 2 kudos
- 8381 Views
- 2 replies
- 0 kudos
Resolved! Delta table version protocol
I do:df.write.format("delta").mode("append").partitionBy("timestamp").option("mergeSchema", "true").save(destination)If table doesn't exist, it creates new table with "minReaderVersion":3,"minWriterVersion":7.Yesterday it was creating table with "min...
- 8381 Views
- 2 replies
- 0 kudos
- 0 kudos
hello how did you fix this explicitly?how did you enable/disable the auto-enable deletion vectors setting to write again with minReaderVersion 1 and minWriterVersion 2?
- 0 kudos
- 2368 Views
- 2 replies
- 1 kudos
Do I need many wheels for each job in project?
I have a project witch my commons, like sparksession object (to run code in pycharm using databricks connect library and the same code directly on databricks).I have under src a few packages from which DAB creates separate jobs. I'm using PyCharm. S...
- 2368 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi, I hope it would be usefuel. Here are my files: project structure -> DAB_project_structure.pngeach yml file for job definitions -> task_group_1_job.png and task_group_2_job.pngEach .py file has main() method.setup.py:description="wheel file based ...
- 1 kudos
- 1652 Views
- 2 replies
- 0 kudos
how to install the package using --index-url
Hi community,I created a job using databricks asset bundle, but I'm worrying about how to install this dependency in the right way?because, I was testing the related job, but seems it doesn't install the torch library properly
- 1652 Views
- 2 replies
- 0 kudos
- 0 kudos
I tried to do it manually and it works.. through databricks asset bundle no. But, I did at the end: dependencies: - torch==2.5.1 - --index-url https://download.pytorch.org/whl/cpu It says:Error: file doesn't exi...
- 0 kudos
- 9498 Views
- 4 replies
- 1 kudos
Unexpected error while calling Notebook string matching regex `\$[\w_]+' expected but `M' found
Run result unavailable: job failed with error message INVALID_PARAMETER_VALUE: Failed to parse %run command: string matching regex `\$[\w_]+' expected but `M' found) Stacktrace:/Notebookpath: scalato call notebook we are using dbutils.notebook.run("N...
- 9498 Views
- 4 replies
- 1 kudos
- 1 kudos
Since you're using dbutils.notebook.run() properly now, the issue is not in your current notebook, but actually in the target notebook you’re calling.Specifically, Databricks is trying to parse a %run command in that notebook, and it’s hitting a synt...
- 1 kudos
- 773 Views
- 1 replies
- 0 kudos
Blade bridge Analyzer out of memory issue
We are running bladebridge analyzer, and we are getting to run out of memorywe tried to increase the RAM and still it gives the same error.We cannot run the analyzer against subset of metadata as it would not generate comprehensive report with how th...
- 773 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Vasu_Kumar_T,How are you doing today?, As per my understanding, running out of memory with BladeBridge Analyzer can be tough, especially when you're working with large and complex metadata where you need the full picture. Even if you've increased ...
- 0 kudos
- 1085 Views
- 1 replies
- 1 kudos
Medaillon architecture
Hello, I have in a S3 data lake, in it: a structure of files that are of different formats : json, csv, text, binary, ...Would you consider this as my bronze layer ? or a "pre-bronze" layer since it can't be processed directly by spark (because of d...
- 1085 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi patacoing,How are you doing today?, As per my understanding, The structure you described in your S3 data lake sounds more like a "pre-bronze" layer, since the files are in mixed formats (JSON, CSV, text, binary), which makes it tricky to process t...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
