- 3296 Views
- 1 replies
- 0 kudos
Spark Driver failed due to DRIVER_UNAVAILABLE but not due to memory pressure
Hello,I have a job cluster running streaming job and it unexpectedly failed on 19th March due to DRIVER_UNAVAILABLE (Request timed out, Driver is temporarily unavailable) in event log. This is the run: https://atlassian-discover.cloud.databricks.com/...
- 3296 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @duliu , Hope you are doing well! Would you kindly see if the KB article below addresses your problem? https://kb.databricks.com/en_US/jobs/driver-unavailable Please let me know if this helps and leave a like if this information is useful, followu...
- 0 kudos
- 2284 Views
- 0 replies
- 0 kudos
Issue while writing data to unity catalog using JDBC
While writing the data to a pre-existing table in the unity catalog using JDBC. it just writes the Delta of the data. Driver used: com.databricks:databricks-jdbc:2.6.36Lets say I have the table has rows:+-+-+ |a|b| +-+-+ |1|2| |3|4| and I am appendi...
- 2284 Views
- 0 replies
- 0 kudos
- 1807 Views
- 0 replies
- 0 kudos
[Delta Sharing - open sharing protocol] Token rotation
Hi, Do you have any experience of rotating Tokens in Delta Sharing automatically?There is an option to do that using CLI (Create and manage data recipients for Delta Sharing | Databricks on AWS). But what to do next? Sending new link to the token via...
- 1807 Views
- 0 replies
- 0 kudos
- 2656 Views
- 1 replies
- 1 kudos
Materialized View to External Location
As I learned the Materialized View is actually a Delta Table stored internally to Databricks (managed table ?)Is it possible to move the location of the Materialized View and the Delta Table under hood to an external location like BLOB?
- 2656 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi @QPeiran For now there is no way to move the materialized view as it's being maintained by Databricks only
- 1 kudos
- 7500 Views
- 3 replies
- 0 kudos
The spark context has stopped and the driver is restarting. Your notebook will be automatically
I am trying to execute a scala jar in notebook. When I execute it explicitly I am able to run the jar like this :but when I am trying to run a notebook through databricks workflow I get the below error : The spark context has stopped and the driver i...
- 7500 Views
- 3 replies
- 0 kudos
- 0 kudos
Could you share the code you have in your JAR file? how are you creating your Spark context in your JAR file?
- 0 kudos
- 4141 Views
- 0 replies
- 0 kudos
JDBC drivers for Microsoft Dataverse IO
I want to run Databricks ETLs on on-prem Unix, Azure and on AWS (in future). I am trying to find suitable JDBC drivers but couldn't find anything except CDATA which is very costly.Can someone please help me? Also, what could be other viable solutions...
- 4141 Views
- 0 replies
- 0 kudos
- 12840 Views
- 7 replies
- 3 kudos
Resolved! In Azure Databricks CLI, how to pass in the parameter notebook_params? Error: Got unexpected extra argument
I am trying to call run-now with notebook_params in Azure Databricks CLI, following https://learn.microsoft.com/en-us/azure/databricks/dev-tools/cli/jobs-cliandescapse the quotes as stated in the documentationhttps://learn.microsoft.com/en-us/azure/d...
- 12840 Views
- 7 replies
- 3 kudos
- 3 kudos
I have the latest Databricks CLI setup and configured in my Ubuntu VM. When I tried to run a job using the json template I generated using databricks jobs get 'xxxjob_idxxx' > orig.json it throws an unknown error.Databricks CLI v0.216.0databricks job...
- 3 kudos
- 12686 Views
- 1 replies
- 1 kudos
Hive vs Delta
I'm curious about your experiences with Hive and Delta Lake. What are the advantages of using Delta over Hive, and in what scenarios would you recommend choosing Delta for data processing tasks? I'd appreciate any insights or recommendations based on...
- 12686 Views
- 1 replies
- 1 kudos
- 1 kudos
Delta Lake offers several advantages over Hive. One of the key benefits is its design for petabyte-scale data lakes with streaming and fast access at the forefront. This makes it more suitable for near-real-time streams, unlike Hive. Delta Lake also ...
- 1 kudos
- 3890 Views
- 2 replies
- 0 kudos
Drop array in a struct field
Hi guys,look my table definitionwell, I need to remove 'med array' inside that 'equip' field.have any idea ?Thank you
- 3890 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi William,There is array_remove method that can help to remove the elements from an array. Here med array is an element in equip array.If it not helpful, please share some sample data so that we can solve it together.Reference: array_removeThanks
- 0 kudos
- 4571 Views
- 2 replies
- 1 kudos
Resolved! Getting errors while reading data from URL
I'm encountering some issues while trying to read a public dataset from a URL using Databricks. Here's the code snippet(along with errors) I'm working with: I'm confused about Delta format error here.When I read data from a URL, how would it have a D...
- 4571 Views
- 2 replies
- 1 kudos
- 1 kudos
@sharma_kamal Please disable the formatCheck in notebook and check if you could read the data The configuration command %sql SET spark.databricks.delta.formatCheck.enabled=false will disable the format check for Delta tables in Databricks. Databrick...
- 1 kudos
- 3332 Views
- 2 replies
- 1 kudos
Can I use Git provider with using Service Principal in job
Hi everyone,I'm trying to use Git provider in Databricks job.First, I was using my personal user account to `Run as`.But when I change `Run as` to Service Principal, it was failed because of permission error.And I can't find a way to solve it.Could I...
- 3332 Views
- 2 replies
- 1 kudos
- 1 kudos
The documentation is lacking in this area which should be easy to set up. Instead we are forced to search among community topics such as these.
- 1 kudos
- 5032 Views
- 0 replies
- 0 kudos
[ Databricks - Delta sharing ] Issue with Delta Sharing in Databricks: Unable to Query Shared Views
Hi guys, I've encountered an issue while attempting to query shared views via Delta Sharing in Databricks. We are using Delta Sharing Databricks-to-Databricks protocol to share data from a databricks environment deployed on azure to another databrick...
- 5032 Views
- 0 replies
- 0 kudos
- 2276 Views
- 0 replies
- 0 kudos
Can't run Delta Live Tables pipeline while using Mosaic
I'm just learning to use mosaic and delta live table in Databricks. I was following this example https://github.com/databrickslabs/mosaic/tree/main/notebooks/examples/python/OpenStreetMaps and after configuring everything I'm not able to run the pipe...
- 2276 Views
- 0 replies
- 0 kudos
- 1736 Views
- 0 replies
- 0 kudos
Delta Sharing - RESOURCE_LIMIT_EXCEEDED (AddFiles, RemoveFiles)
Hi,I would like to start using Delta Sharing but I would need to check if my tables hit limitations which are in the following article: RESOURCE_LIMIT_EXCEEDED error when querying a Delta Sharing table - Databricks.Delta Sharing has limits on the met...
- 1736 Views
- 0 replies
- 0 kudos
- 2859 Views
- 3 replies
- 0 kudos
Unable to create/save job of type "python script"
Hi All,We are facing an issue while creating a simple job of type "python script". A python file in workspace is selected as a source. No arguments/job parameters are provided. This is a strange behavior and just started occurring since today morning...
- 2859 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi Ayushi,How can I call call RESET API? this issue is occurring when creating a new job from databricks web UI. It looks like REST API is for resetting job settings of an existing job.Can this be an issue with the databricks workspace we are using?A...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
