- 876 Views
- 2 replies
- 0 kudos
DAB for_each_task python wheel fail
using python_wheel_wrapper experimental true allows me to use python_wheel_task on an older cluster.However, if I embed the python_wheel_task in a for_each_task it fails at runtime with: "Library installation failed for library due to user error. Er...
- 876 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @BillBishop, I will check on this internally as outcome does not seem to be correct. If possible, upgrade your cluster to DBR 14.1 or later. This would resolve the issue without relying on the experimental feature
- 0 kudos
- 5531 Views
- 5 replies
- 0 kudos
sparkContext in Runtime 15.3
Hello All, Our Azure databricks cluster is running under "Legacy Shared Compute" policy with 15.3 runtime. One of the python notebooks is used to connect to an Azure SQL database to read/insert data. The following snippet of code is responsible for r...
- 5531 Views
- 5 replies
- 0 kudos
- 0 kudos
@rushi29 @GangsterI ended up implementing pyodbc with the mssql driver using init scripts.Spark context is no longer usable in shared compute so that was the only approach we could take.
- 0 kudos
- 5090 Views
- 7 replies
- 1 kudos
Resolved! databricks workspace import_dir not working without any failure message
Morning everyone!I`m trying to copy from the repo into the databricks workspace all the notebooks. I`m using the command: databricks workspace import_dir . /Shared/Notebooks, it will just print all the info regarding the Workspace API.If I launch dat...
- 5090 Views
- 7 replies
- 1 kudos
- 1 kudos
Hi all,I`ve disinstalled and installed again databricks-cli and now worked.Is not a real solution but still it worked after one week...
- 1 kudos
- 764 Views
- 1 replies
- 0 kudos
Autoloader is not able to infer schema from json
Hi, I have json files and it contains json array and only one object (payload below), I have set in autoloader inferSchema to true, however autoloader throws:"Failed to infer schema for format json from existing files ..."I have also check option to ...
- 764 Views
- 1 replies
- 0 kudos
- 0 kudos
Yep, my mistake, json file was corrupted.
- 0 kudos
- 1700 Views
- 2 replies
- 0 kudos
External connection to Azure ADLS Gen2 storage
Hello, I have a problem trying to make an external connection to a blob storage configured as ADLS Gen2 with hierarchical namespace (HNS) enabled. I have setuup the storage account with a container wirh HNS enabled as in the image attached:Next I hav...
- 1700 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi, would it work if you try to remove "landing" at the end? Seems the error is complaining that the external location should be a directory.
- 0 kudos
- 1757 Views
- 1 replies
- 0 kudos
Resolved! Hostname not resolving using Spark JDBC
Hey guys,Ive ran into a weird error this morning. Last week I was testing a new Oracle Connector and it ran through smooth the whole last week!This morning at 7 it ran again and following it was showing a "SQLRecoverableException: IO Error: Unknown h...
- 1757 Views
- 1 replies
- 0 kudos
- 0 kudos
I have even cloned the cluster and it worked on the new one. But after turning the cluster off over night, it started working again the next morning. This is really weird.
- 0 kudos
- 1221 Views
- 4 replies
- 0 kudos
Does DBR 14.3 not support Describe history command?
Hello, We have recently updated DBR version from 9.1 LTS to 14.3 LTS and observed that DESCRIBE HISTORY is not supported in 14.3 LTS. Could you please suggest any alternative to be used for table history?
- 1221 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi, I'm still not able to recreate this issue with Standard_DS3_v2. I'm not sure if this is relevant, but do you also have this issue on an old High Concurrency cluster with custom access mode for the Standard_DS3_v2 cluster too?
- 0 kudos
- 737 Views
- 1 replies
- 1 kudos
External Location Naming Issue & Impact of Renaming in Unity Catalog
Hey,I created an external location in my test environment using a mount point . Now, when I try to create the same external location in prod, it doesn’t allow me to use the same name. Is there any specific reason for this restriction in Unity Catalog...
- 737 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @Faizan_khan8171, Thanks for your question. In Unity Catalog, external location names must be unique within the metastore. This restriction prevents naming conflicts and ensures that every external location is easily identifiable and manageable...
- 1 kudos
- 2887 Views
- 6 replies
- 0 kudos
Move Databricks service to another resource group
Hello,Is it possible to move in another resource group the databricks service without any problem?I have a resource group where there are two workspaces the prod and staging environment, I created another resource group to maintain only the databrick...
- 2887 Views
- 6 replies
- 0 kudos
- 0 kudos
I'm running into the same problem, what's the procedure to create a feature request for this? It seems to me that when DB is running in Azure that I should be able to move it to a different resource group.
- 0 kudos
- 15728 Views
- 3 replies
- 1 kudos
Failed to create new KafkaAdminClient
I want to create connections to kafka with spark.readStream using the following parameters: kafkaParams = { "kafka.sasl.jaas.config": f'org.apache.kafka.common.security.plain.PlainLoginModule required username="{kafkaUsername}" password="{kafkaPa...
- 15728 Views
- 3 replies
- 1 kudos
- 1 kudos
If you are using Confluent with Schema Registry you can use the below code. No additional libraries need to be installed. From Databricks Runtime 16.0 it support schema references and recursive references: from pyspark.sql.functions import col, lit f...
- 1 kudos
- 6892 Views
- 2 replies
- 4 kudos
JDBC Error: Error occured while deserializing arrow data
I am getting the following error in my Java application.java.sql.SQLException: [Databricks][DatabricksJDBCDriver](500618) Error occured while deserializing arrow data: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not availableI beli...
- 6892 Views
- 2 replies
- 4 kudos
- 4 kudos
For anyone encountering this issue in 2025, I was able to solve it by using the --add-opens=jdk.unsupported/sun.misc=ALL-UNNAMEDoption in combination with the latest jdbc driver (v2.7.1). I was using the driver in dbeaver, but I assume the issue coul...
- 4 kudos
- 1442 Views
- 2 replies
- 0 kudos
Resolved! Writing back from notebook to blob storage as single file with UC configured databricks
I want to write a file from notebook to blob storage. we have configured unity catalog. When it writes it creates the folder name as the file name that I have provided and inside that it writes multiple files as show below. Can someone suggest me on ...
- 1442 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi ShivapIf you want to save a dataframe as a single file, you could consider to convert the pyspark dataframe to a pandas dataframe and then save it as file. path_single_file = '/Volumes/demo/raw/test/single' # create sample dataframe df = spark.cr...
- 0 kudos
- 14873 Views
- 6 replies
- 4 kudos
Read external iceberg table in a spark dataframe within databricks
I am trying to read an external iceberg database from s3 location using the follwing commanddf_source = (spark.read.format("iceberg") .load(source_s3_path) .drop(*source_drop_columns) .filter(f"{date_column}<='{date_filter}'") )B...
- 14873 Views
- 6 replies
- 4 kudos
- 878 Views
- 1 replies
- 0 kudos
Resolved! How to use GLOW in Databricks Premium on AWS?
Hi!Have connected workspace to AWS, but when I execute in a new notebook: %python %pip install glow.py import glow from pyspark.sql import SparkSession # Create a Spark session spark = (SparkSession.builder .appName("Genomics Analysis") ...
- 878 Views
- 1 replies
- 0 kudos
- 0 kudos
Solved this with the help of colleagues at last. First of all, it won't work with Serverless mode, so a cluster is required. Once the cluster is created in Compute section, on Library tab add those 2 libraries:Then running:import glow from pyspark.sq...
- 0 kudos
- 2045 Views
- 2 replies
- 0 kudos
Python callback functions fail to trigger
How can I get sys.exceptionhook and atexit module to trigger a callback function on exit of a python notebook? These fail to work when an unhandled exception is encountered (exceptionhook), or the program exits (atexit).
- 2045 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey Lozik,Ran into this myself as well. The reason this doesn't work is because Databricks is using Ipython under the hood.The following codesnippet creates an exception hook for all exceptions (using the general Exception), it's also possible to spe...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 554 | |
| 349 | |
| 287 |
