cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

alejandrofm
by Valued Contributor
  • 2825 Views
  • 10 replies
  • 15 kudos

All-purpose clusters not remembering custom tags

Hi, we have several clusters used with Notebooks, we don't delete them, just start-stop according to the "minutes of inactivity" set.I'm trying to set a custom tag, so I wait until the cluster shuts down, add a tag, check that the tag is among then "...

  • 2825 Views
  • 10 replies
  • 15 kudos
Latest Reply
Dribka
New Contributor III
  • 15 kudos

@alejandrofm the behavior you're describing, where the custom tag disappears after the cluster restarts, might be related to the cluster configuration or the specific settings of your Databricks environment. To troubleshoot this, ensure that the cust...

  • 15 kudos
9 More Replies
liamod_1
by New Contributor III
  • 24044 Views
  • 11 replies
  • 9 kudos

Resolved! Failure starting repl

Hi, we have several clusters that keep giving this error:Failure starting repl. Try detaching and re-attaching the notebook.All the investigation I've done points to this issue being related to the number of concurrent connections but we only have 1 ...

  • 24044 Views
  • 11 replies
  • 9 kudos
Latest Reply
Kareemlowe46
New Contributor II
  • 9 kudos

After some initial skepticism, Barker agreed to give Plinko https://plnkgame.com a try. The game was an instant hit with both the audience and the contestants. The concept was simple but exciting - players would drop a disc down a large pegboard, and...

  • 9 kudos
10 More Replies
bshirdi
by New Contributor II
  • 1237 Views
  • 1 replies
  • 2 kudos

Getting HTTP 502 bad gateway error!

Hello all,I am suddenly getting an HTTP 502 and DRIVER_LIBRARY_INSTALLATION_FAILURE error during the Python library installation when the cluster gets initialized. I have around 10 Python packages out of which 2-3, packages always failed to install a...

image.png
  • 1237 Views
  • 1 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

Hi @Bhargav Shir​ Great to meet you, and thanks for your question! Let's see if your peers in the community have an answer to your question. Thanks.

  • 2 kudos
Inna_M
by New Contributor III
  • 873 Views
  • 1 replies
  • 1 kudos

Resolved! Is there any maintenance (patches , upgrade for VMs created by DataBricks on Azure) from DataBricks

We are using Databricks on Azure. Infra team noticed we have some VMs created in the past for DataBricks clusters on version Linux (ubuntu 18.04). Is there maintenance previewed for that, upgrade? Are there any patches for created in Azure objects by...

  • 873 Views
  • 1 replies
  • 1 kudos
Latest Reply
Inna_M
New Contributor III
  • 1 kudos

Finally while I was posting this question, AzureDataBricks upgraded VMs to the supported version 20, not the latest , 22. It was a week after old version was no longer supported by Microsoft

  • 1 kudos
akc
by New Contributor III
  • 1045 Views
  • 3 replies
  • 5 kudos

Resolved! Training models on big or small clusters

I have a workflow with a model which trains every sunday in Azure Databricks. Sometimes the workflow fails as the max wait time is exceeded (currently I am using 1200 seconds). To solve the problem I was thinking of either increasing the wait time or...

  • 1045 Views
  • 3 replies
  • 5 kudos
Latest Reply
Anonymous
Not applicable
  • 5 kudos

Hi @Andreas Kaae​ Thank you for posting your question in our community! We are happy to assist you.To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers ...

  • 5 kudos
2 More Replies
vnc001
by New Contributor
  • 779 Views
  • 1 replies
  • 1 kudos

Resolved! Clusters API 2.0 - Unable to execute cluster events api

Details: I keep getting "Missing required field: cluster_id" even though you can see it is supplied. Is this a bug? or I am missing something? I am testing this in postman. Error: {"error_code":"INVALID_PARAMETER_VALUE","message":"Missing required fi...

image
  • 779 Views
  • 1 replies
  • 1 kudos
Latest Reply
SUMI1
New Contributor III
  • 1 kudos

Hi guysI'm sorry to hear that the Clusters API 2.0 and cluster event execution are giving you trouble. I advise getting in touch with the support staff for guidance on quickly fixing the problem.

  • 1 kudos
J_
by New Contributor II
  • 6342 Views
  • 7 replies
  • 6 kudos

Resolved! Clusters stuck on pending indefinitely (community edition)

From yesterday, suddenly clusters do not start and are in the pending state indefinitely (more than 30 minutes). From a previous post, I tried to add 443 port to the firewall but it doesn't help. In the clusters page, the message says: Finding instan...

  • 6342 Views
  • 7 replies
  • 6 kudos
Latest Reply
Reet
New Contributor II
  • 6 kudos

I am also having same issue and there seems to be no outage.....

  • 6 kudos
6 More Replies
Leodatabricks
by Contributor
  • 2547 Views
  • 14 replies
  • 23 kudos

How to secure all clusters and then start running the code

When there are slow nodes, sometimes a job needs to resize its number of clusters to reach the required number of nodes. Is there any way to make sure no code is running before all nodes are secured? Thank you!

  • 2547 Views
  • 14 replies
  • 23 kudos
Latest Reply
Anonymous
Not applicable
  • 23 kudos

Hi @Leo Bao​ Hope everything is going great.Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we ca...

  • 23 kudos
13 More Replies
Ajay-Pandey
by Esteemed Contributor II
  • 514 Views
  • 1 replies
  • 6 kudos

Cluster policies now support limiting the max number of clusters per user can create Policy permissions allow you to set a max number of clusters per ...

Cluster policies now support limiting the max number of clusters per user can createPolicy permissions allow you to set a max number of clusters per user. This determines how many clusters a user can create using that policy. If the user exceeds the ...

image
  • 514 Views
  • 1 replies
  • 6 kudos
Latest Reply
jose_gonzalez
Moderator
  • 6 kudos

Thank you for sharing

  • 6 kudos
antoooks
by New Contributor III
  • 3828 Views
  • 6 replies
  • 10 kudos

Resolved! Databricks clusters stuck on Pending and Terminating state indefinitely

Hi everyone,Our company is using Databricks on GKE. It works fine until suddenly when we try to create and terminate clusters today, it got stuck on Pending and Terminating state for hours (now more than 6 hours). There is no conclusion can be drawn ...

screenshot
  • 3828 Views
  • 6 replies
  • 10 kudos
Latest Reply
Databricks_Buil
New Contributor III
  • 10 kudos

Hi @Kurnianto Trilaksono Sutjipto​ : Figured out after multiple connects that This is typically a cloud provider issue. You can file a support ticket if the issue persists.

  • 10 kudos
5 More Replies
Aviral-Bhardwaj
by Esteemed Contributor III
  • 586 Views
  • 0 replies
  • 31 kudos

Understanding Cluster Pools Sometimes we want to run our databricks code without any delay as reports are very emergency like the upstream team wants ...

Understanding Cluster PoolsSometimes we want to run our databricks code without any delay as reports are very emergency like the upstream team wants to save as much time as they can save in the starting cluster.That time we can use the pool of cluste...

  • 586 Views
  • 0 replies
  • 31 kudos
Sujitha
by Community Manager
  • 636 Views
  • 1 replies
  • 4 kudos

Documentation Update  Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, and data enginee...

Documentation Update Databricks documentation provides how-to guidance and reference information for data analysts, data scientists, and data engineers working in the Databricks Data Science & Engineering, Databricks Machine Learning, and Databricks ...

  • 636 Views
  • 1 replies
  • 4 kudos
Latest Reply
Harun
Honored Contributor
  • 4 kudos

Thanks for sharing @Sujitha Ramamoorthy​ 

  • 4 kudos
NicolasEscobar
by New Contributor II
  • 6205 Views
  • 8 replies
  • 5 kudos

Resolved! Job fails after runtime upgrade

I have a job running with no issues in Databricks runtime 7.3 LTS. When I upgraded to 8.3 it fails with error An exception was thrown from a UDF: 'pyspark.serializers.SerializationError'... SparkContext should only be created and accessed on the driv...

  • 6205 Views
  • 8 replies
  • 5 kudos
Latest Reply
User16873042682
New Contributor II
  • 5 kudos

Adding to @Sean Owen​  comments, The only reason this is working is that the optimizer is evaluating this locally rather than creating a context on executors and evaluating it.

  • 5 kudos
7 More Replies
MxSasch
by New Contributor II
  • 2971 Views
  • 13 replies
  • 5 kudos

Cluster terminated.Reason:Unexpected launch failure

Is there any known issues affecting the creation of clusters? I've been unable to get any clusters to start today so far!Have received this error "Cluster terminated.Reason:Unexpected launch failure"Help!

  • 2971 Views
  • 13 replies
  • 5 kudos
Latest Reply
AWe
New Contributor II
  • 5 kudos

good

  • 5 kudos
12 More Replies
Nuthan_1994
by New Contributor II
  • 1990 Views
  • 4 replies
  • 3 kudos

Resolved! Installing new libraries on Azure Databricks Clusters

Hi Everyone,I was trying to install the newest python version on the Databricks Clusters and it has the runtime version 7.3 LTS, but no matter how many times I try it keeps installing the 3.7.5 version of python.I know that Runtime version 7.3 LTS co...

  • 1990 Views
  • 4 replies
  • 3 kudos
Latest Reply
Kaniz
Community Manager
  • 3 kudos

Hi @Nuthan Peddapurapu​ , Not supported with Databricks Runtime 7 and above at the momenthttps://docs.databricks.com/libraries/cluster-libraries.html#library

  • 3 kudos
3 More Replies
Labels