- 6279 Views
- 10 replies
- 15 kudos
All-purpose clusters not remembering custom tags
Hi, we have several clusters used with Notebooks, we don't delete them, just start-stop according to the "minutes of inactivity" set.I'm trying to set a custom tag, so I wait until the cluster shuts down, add a tag, check that the tag is among then "...
- 6279 Views
- 10 replies
- 15 kudos
- 15 kudos
@alejandrofm the behavior you're describing, where the custom tag disappears after the cluster restarts, might be related to the cluster configuration or the specific settings of your Databricks environment. To troubleshoot this, ensure that the cust...
- 15 kudos
- 7825 Views
- 1 replies
- 2 kudos
Getting HTTP 502 bad gateway error!
Hello all,I am suddenly getting an HTTP 502 and DRIVER_LIBRARY_INSTALLATION_FAILURE error during the Python library installation when the cluster gets initialized. I have around 10 Python packages out of which 2-3, packages always failed to install a...
- 7825 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Bhargav Shir​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.
- 2 kudos
- 49904 Views
- 8 replies
- 8 kudos
Resolved! Failure starting repl
Hi, we have several clusters that keep giving this error:Failure starting repl. Try detaching and re-attaching the notebook.All the investigation I've done points to this issue being related to the number of concurrent connections but we only have 1 ...
- 49904 Views
- 8 replies
- 8 kudos
- 8 kudos
@Aviral Bhardwaj​ thanks, this seemed to fix the issue, we had an innit script that was potentially conflicting with UI set libraries (in cluster settings).
- 8 kudos
- 1612 Views
- 1 replies
- 1 kudos
Resolved! Is there any maintenance (patches , upgrade for VMs created by DataBricks on Azure) from DataBricks
We are using Databricks on Azure. Infra team noticed we have some VMs created in the past for DataBricks clusters on version Linux (ubuntu 18.04). Is there maintenance previewed for that, upgrade? Are there any patches for created in Azure objects by...
- 1612 Views
- 1 replies
- 1 kudos
- 1 kudos
Finally while I was posting this question, AzureDataBricks upgraded VMs to the supported version 20, not the latest , 22. It was a week after old version was no longer supported by Microsoft
- 1 kudos
- 2557 Views
- 3 replies
- 5 kudos
Resolved! Training models on big or small clusters
I have a workflow with a model which trains every sunday in Azure Databricks. Sometimes the workflow fails as the max wait time is exceeded (currently I am using 1200 seconds). To solve the problem I was thinking of either increasing the wait time or...
- 2557 Views
- 3 replies
- 5 kudos
- 5 kudos
Hi @Andreas Kaae​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...
- 5 kudos
- 1529 Views
- 1 replies
- 1 kudos
Resolved! Clusters API 2.0 - Unable to execute cluster events api
Details: I keep getting "Missing required field: cluster_id" even though you can see it is supplied. Is this a bug? or I am missing something? I am testing this in postman. Error: {"error_code":"INVALID_PARAMETER_VALUE","message":"Missing required fi...
- 1529 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi guysI'm sorry to hear that the Clusters API 2.0 and cluster event execution are giving you trouble. I advise getting in touch with the support staff for guidance on quickly fixing the problem.
- 1 kudos
- 9060 Views
- 6 replies
- 5 kudos
Resolved! Clusters stuck on pending indefinitely (community edition)
From yesterday, suddenly clusters do not start and are in the pending state indefinitely (more than 30 minutes). From a previous post, I tried to add 443 port to the firewall but it doesn't help. In the clusters page, the message says: Finding instan...
- 9060 Views
- 6 replies
- 5 kudos
- 5853 Views
- 14 replies
- 23 kudos
How to secure all clusters and then start running the code
When there are slow nodes, sometimes a job needs to resize its number of clusters to reach the required number of nodes. Is there any way to make sure no code is running before all nodes are secured? Thank you!
- 5853 Views
- 14 replies
- 23 kudos
- 23 kudos
Hi @Leo Bao​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ca...
- 23 kudos
- 1114 Views
- 1 replies
- 6 kudos
Cluster policies now support limiting the max number of clusters per user can create Policy permissions allow you to set a max number of clusters per ...
Cluster policies now support limiting the max number of clusters per user can createPolicy permissions allow you to set a max number of clusters per user. This determines how many clusters a user can create using that policy. If the user exceeds the ...
- 1114 Views
- 1 replies
- 6 kudos
- 6 kudos
Thank you for sharing
- 6 kudos
- 6440 Views
- 5 replies
- 10 kudos
Resolved! Databricks clusters stuck on Pending and Terminating state indefinitely
Hi everyone,Our company is using Databricks on GKE. It works fine until suddenly when we try to create and terminate clusters today, it got stuck on Pending and Terminating state for hours (now more than 6 hours). There is no conclusion can be drawn ...
- 6440 Views
- 5 replies
- 10 kudos
- 10 kudos
Hi @Kurnianto Trilaksono Sutjipto​ : Figured out after multiple connects that This is typically a cloud provider issue. You can file a support ticket if the issue persists.
- 10 kudos
- 1162 Views
- 0 replies
- 31 kudos
Understanding Cluster Pools Sometimes we want to run our databricks code without any delay as reports are very emergency like the upstream team wants ...
Understanding Cluster PoolsSometimes we want to run our databricks code without any delay as reports are very emergency like the upstream team wants to save as much time as they can save in the starting cluster.That time we can use the pool of cluste...
- 1162 Views
- 0 replies
- 31 kudos
- 1171 Views
- 1 replies
- 4 kudos
Documentation Update  Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, and data enginee...
Documentation Update Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, and data engineers working in the Databricks Data Science & Engineering, Databricks Machine Learning, and Databricks ...
- 1171 Views
- 1 replies
- 4 kudos
- 4 kudos
Thanks for sharing @Sujitha Ramamoorthy​
- 4 kudos
- 8787 Views
- 7 replies
- 5 kudos
Resolved! Job fails after runtime upgrade
I have a job running with no issues in Databricks runtime 7.3 LTS. When I upgraded to 8.3 it fails with error An exception was thrown from a UDF: 'pyspark.serializers.SerializationError'... SparkContext should only be created and accessed on the driv...
- 8787 Views
- 7 replies
- 5 kudos
- 5 kudos
Adding to @Sean Owen​ comments, The only reason this is working is that the optimizer is evaluating this locally rather than creating a context on executors and evaluating it.
- 5 kudos
- 6627 Views
- 11 replies
- 5 kudos
Cluster terminated.Reason:Unexpected launch failure
Is there any known issues affecting the creation of clusters? I've been unable to get any clusters to start today so far!Have received this error "Cluster terminated.Reason:Unexpected launch failure"Help!
- 6627 Views
- 11 replies
- 5 kudos
- 3656 Views
- 4 replies
- 3 kudos
Resolved! uninstalled libraries continue to get installed on cluster startup
We have been trying to update some library versions by uninstalling the old versions and installing new ones. However, the old libraries continue to get installed on cluster startup despite not showing up in the "libraries" tab of the cluster page. W...
- 3656 Views
- 4 replies
- 3 kudos
- 3 kudos
The issue seemed to go away on its own. At some point the libraries page started showing what was getting installed to the cluster, and removing libraries from the page caused them to stop getting installed on cluster startup. I'm guessing there was ...
- 3 kudos