- 19878 Views
- 5 replies
- 2 kudos
Resolved! Error: The spark driver has stopped unexpectedly and is restarting. Your notebook will be automatically reattached.
What is the problem?I am getting this error every time I run a python notebook on my Repo in Databricks.BackgroundThe notebook where I am getting the error is a notebook that creates a dataframe and the last step is to write the dataframe to a Delta ...
- 19878 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Sara Corral Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
- 2 kudos
- 4079 Views
- 4 replies
- 4 kudos
Unable to use dbutils in Premium
I am unable to use dbutils commands and mkdir, etc also does not work after upgrading my Databricks Workspace from Standard tier to Premium tier.It throws the following error:py4j.security.Py4JSecurityException: Constructor public com.databricks.back...
- 4079 Views
- 4 replies
- 4 kudos
- 4 kudos
Hi @Abhishek Jain Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Than...
- 4 kudos
- 6460 Views
- 2 replies
- 2 kudos
Resolved! AZURE_QUOTA_EXCEEDED_EXCEPTION - even with more than vCPUs than Databricks recommends
I am running this Delta Live Tables PoC from databricks-industry-solutions/industry-solutions-blueprintshttps://github.com/databricks-industry-solutions/pos-dltI have Standard_DS4_v2 with 28GB and 8 cores x 2 workers - so a total of 16 cores. This is...
- 6460 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi @Prasenjit Biswas We haven't heard from you since the last response from @Jose Gonzalez . Kindly share the information with us, and in return, we will provide you with the necessary solution.Thanks and Regards
- 2 kudos
- 6143 Views
- 12 replies
- 0 kudos
Resolved! Can't use job cluster for scheduled jobs ADD_NODES_FAILED : Failed to add 9 containers to the cluster. Will attempt retry: false. Reason: Azure Quota Exceeded Exception
Hi everyone,I've been using my all purpose cluster for scheduled jobs and I've been told that it's a suboptimal thing to do and that using a job cluster for the scheduled jobs cuts costs by half.Unfortunately, when I tried to switch clusters on my ex...
- 6143 Views
- 12 replies
- 0 kudos
- 0 kudos
@Bassem Jaber If you are seeing same error then you need to increase quota, for that your azure plan should be changed from pay as you go to other plan. as pay-as-go azure model has limitations on quota increase
- 0 kudos
- 4090 Views
- 2 replies
- 1 kudos
Resolved! ImportError: cannot import name dataclass_transform
Hi,I am using Standard Runtime 11.3 LTS and trying to utilize spacy-> en_core_web_sm but I am getting following error.ImportError: cannot import name dataclass_transformIt was working last week but stopped working recently.Appreciate any help. Regard...
- 4090 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @Sanjay Jain Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks...
- 1 kudos
- 2249 Views
- 3 replies
- 1 kudos
Error when creating a delta live table pipeline
INVALID_PARAMETER_VALUE: Validation failed for node_type_id, the value must be Standard_DS3_v2 (is "Standard_F8s")
- 2249 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Nitya Mehta Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers y...
- 1 kudos
- 3737 Views
- 5 replies
- 0 kudos
How can I build a custom docker image for the ML runtime (e.g. 12.1 ML)?
I successfully built a custom docker image for the Standard runtime following the steps described on the page Customize containers with Databricks Container Services and based on the image databricksruntime/standard:11.3-LTS. However, I cannot find ...
- 3737 Views
- 5 replies
- 0 kudos
- 0 kudos
Hi @Tibor Fabian Help us build a vibrant and resourceful community by recognizing and highlighting insightful contributions. Mark the best answers and show your appreciation!
- 0 kudos
- 2842 Views
- 3 replies
- 0 kudos
Resolved! Why does my Single Node cluster automatically resize num_workers?
Hi community,We have setup a Databricks cluster as Single node with num_workers=0 . Sometimes the cluster automatically resizes to e.g. 10 workers. When I edit the cluster subsequently it gives an error that num_workers is not allowed for Single node...
- 2842 Views
- 3 replies
- 0 kudos
- 0 kudos
I think the issue is solved, at least it didn't occur in the last month. We monitored this via Azure Log Analytics. Maybe it was solved due to some patch/update, thanks anyway!
- 0 kudos
- 4301 Views
- 5 replies
- 2 kudos
Resolved! Facing issue While executing DDL and DML queries in 12.0 cluster runtime version.
Hi all,Currently we are using Driver: Standard_D32s_v3 · Workers: Standard_D32_v3 · 2-8 workers · 6.4 Extended Support (includes Apache Spark 2.4.5, Scala 2.11) cluster. For this we are running 24/7 streaming notebook on trigger of every minute and 5...
- 4301 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @Someswara Durga Prasad Yaralgadda Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love t...
- 2 kudos
- 8014 Views
- 9 replies
- 3 kudos
Resolved! Can't write / overwrite delta table with error: oxxxx.saveAsTable. (Driver Error: OutOfMemory)
Current Cluster Config:Standard_DS3_v2 (14GB, 4 Cores) 2-6 workersStandard_DS3_v2 (14GB, 4Cores) for driverRuntime: 10.4x-scala2.12We want to overwrite a temporary delta table with new records. The records will be load by another delta table and tran...
- 8014 Views
- 9 replies
- 3 kudos
- 3 kudos
Hi,thank you for your help!We tested the configuration settings and it runs without any errors.Could you give us some more information, where we can find some documentation about such settings. We searched hours to fix our problem. So we contacted th...
- 3 kudos
- 3434 Views
- 2 replies
- 0 kudos
Resolved! Pool Max Capacity vs Cluster Max Workers
Hi all, we have a databricks instance on Azure with a Compute Cluster version 7.3 LTS. Currently the cluster has 4 max workers (min workers: 1) of type: Standard_D13_v2 and 1 driver of the same type. There are several jobs that are running on this cl...
- 3434 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @EDDatabricks EDDatabricks Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear f...
- 0 kudos
- 6039 Views
- 1 replies
- 5 kudos
java.util.NoSuchElementException: key not found
Hello,We are using a Azure Databricks with Standard DS14_V2 Cluster with Runtime 9.1 LTS, Spark 3.1.2 and Scala 2.12 and facing the below issue frequently when running our ETL pipeline. As part of the operation that is failing there are several joins...
- 6039 Views
- 1 replies
- 5 kudos
- 5 kudos
Hey man,Please use these configuration in your cluster and it will work,spark.sql.storeAssignmentPolicy LEGACYspark.sql.parquet.binaryAsString truespark.speculation falsespark.sql.legacy.timeParserPolicy LEGACYif it wont work let me know what problem...
- 5 kudos
- 2488 Views
- 4 replies
- 0 kudos
FileNotFoundError when running dbx execute
Hi,I'm very new to databricks, this might be a basic question.I can't find a way to run my local python file with databricks successfully. When I run the following `execute` command, I got a FileNotFoundError.`dbx execute --cluster-id=*** --job=Sampl...
- 2488 Views
- 4 replies
- 0 kudos
- 2210 Views
- 3 replies
- 2 kudos
How to deploy a python script with dependencies by dbx?
Hi,I'm quite new here. I'm trying to perform a deployment of python file with dbx command. The file contains libraries to be installed. How may I deploy the file (together with its dependencies) to databricks?Here are the commands I currently run:`db...
- 2210 Views
- 3 replies
- 2 kudos
- 2 kudos
Hi @Di Lin Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. We'd love to hear from you.Thanks!
- 2 kudos
- 6168 Views
- 9 replies
- 7 kudos
How to optimize exporting dataframe to delta file?
Scenario : I have a dataframe that have 5 billion records/rows and 100+ columns. Is there a way to write this in a delta format efficiently. I have tried to export it but cancelled it after 2 hours (write didnt finish) as this processing time is not ...
- 6168 Views
- 9 replies
- 7 kudos
- 7 kudos
Hi @Franco Sia , Just a friendly follow-up. Do you still need help or the above responses help you to find the solution? Please let us know.
- 7 kudos