cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

TheDataDexter
by New Contributor III
  • 2944 Views
  • 4 replies
  • 3 kudos

Resolved! Single-Node cluster works but Multi-Node clusters do not read data.

I am currently working with a VNET injected databricks workspace. At the moment I have mounted a the databricks cluster on an ADLS G2 resource. When running notebooks on a single node that read, transform, and write data we do not encounter any probl...

  • 2944 Views
  • 4 replies
  • 3 kudos
Latest Reply
ellafj
New Contributor II
  • 3 kudos

@TheDataDexter Did you find a solution to your problem? I am facing the same issue

  • 3 kudos
3 More Replies
brickster_2018
by Esteemed Contributor
  • 9773 Views
  • 2 replies
  • 0 kudos
  • 9773 Views
  • 2 replies
  • 0 kudos
Latest Reply
Mooune_DBU
Valued Contributor
  • 0 kudos

It's set as an environment variable called `DATABRICKS_RUNTIME_VERSION`In your init scripts, you just need to add a line to display or save the info (see python example below):import os print("DATABRICKS_RUNTIME_VERSION:",os.environ.get('DATABRICKS_R...

  • 0 kudos
1 More Replies
SaravananPalani
by New Contributor II
  • 20498 Views
  • 8 replies
  • 9 kudos

Is there any way to monitor the CPU, disk and memory usage of a cluster while a job is running?

I am looking for something preferably similar to Windows task manager which we can use for monitoring the CPU, memory and disk usage for local desktop.

  • 20498 Views
  • 8 replies
  • 9 kudos
Latest Reply
hitech88
New Contributor II
  • 9 kudos

Some important info to look in Gangalia UI in CPU, memory and server load charts to spot the problem:CPU chart :User %Idle %High percentage of user % indicates heavy CPU usage in the cluster.Memory chart : Use %Free %Swap % If you see purple line ove...

  • 9 kudos
7 More Replies
supremefist
by New Contributor III
  • 4155 Views
  • 5 replies
  • 2 kudos

Resolved! New spark cluster being configured in local mode

Hi,We have two workspaces on Databricks, prod and dev. On prod, if we create a new all-purpose cluster through the web interface and go to Environment in the the spark UI, the spark.master setting is correctly set to be the host IP. This results in a...

  • 4155 Views
  • 5 replies
  • 2 kudos
Latest Reply
scottb
New Contributor II
  • 2 kudos

I found the same issue when choosing the default cluster setup on first setup that when I went to edit the cluster to add an instance profile, I was not able to save without fixing this. Thanks for the tip

  • 2 kudos
4 More Replies
Pat
by Honored Contributor III
  • 7098 Views
  • 7 replies
  • 18 kudos

Resolved! Cluster Modes - High Concurrency

It took me quite some time to find the option to create a cluster in High Concurrency mode. It was hidden in the new UI.What should be the way to access the data with TAC?What is the equivalent mode to work with TAC ?Does it mean that we are being pu...

image image.png image
  • 7098 Views
  • 7 replies
  • 18 kudos
Latest Reply
Prabakar
Esteemed Contributor III
  • 18 kudos

Thanks. Always happy to help.

  • 18 kudos
6 More Replies
MadelynM
by New Contributor III
  • 3914 Views
  • 3 replies
  • 3 kudos

How do I move existing workflows and jobs running on an all-purpose cluster to a shared jobs cluster?

A Databricks cluster is a set of computation resources that performs the heavy lifting of all of the data workloads you run in Databricks. Databricks provides a number of options when you create and configure clusters to help you get the best perform...

Left navigation bar selecting Data Science & Engineering Left nav Workflows selected Screen Shot 2022-07-05 at 10.24.37 AM Screen Shot 2022-07-05 at 10.24.46 AM
  • 3914 Views
  • 3 replies
  • 3 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 3 kudos

Hi @Madelyn Mullen​ , Thank you for sharing such an excellent and informative post. We hope to see these very often.

  • 3 kudos
2 More Replies
Megan05
by New Contributor III
  • 2248 Views
  • 4 replies
  • 4 kudos

Resolved! Out of Memory/Connection Lost When Writing to External SQL Server from Databricks Using JDBC Connection

I am working on writing a large amount of data from Databricks to an external SQL server using a JDB connection. I keep getting timeout errors/connection lost but digging deeper it appears to be a memory problem. I am wondering what cluster configura...

  • 2248 Views
  • 4 replies
  • 4 kudos
Latest Reply
hotrabattecom
New Contributor II
  • 4 kudos

Thanks for the answer. I am also get in this problem. Hotrabatt

  • 4 kudos
3 More Replies
rbarrero
by New Contributor III
  • 4368 Views
  • 13 replies
  • 7 kudos

Resolved! Error saving changes on Job Cluster

Hello all and thanks.After apply to serving a model, I go to edit corresponding Job Cluster to configure its init_script but when I try to save changes (Confirm and restart) it thrown the following error:Error: Cannot edit cluster 0503-141315-hu3wd4i...

  • 4368 Views
  • 13 replies
  • 7 kudos
Latest Reply
rbarrero
New Contributor III
  • 7 kudos

Sorry for the delay in responding. Finally a partner could fix the problem, he can edit without problems the cluster and add the init_script.Thank you!

  • 7 kudos
12 More Replies
reedzhang
by New Contributor III
  • 2955 Views
  • 6 replies
  • 5 kudos

Resolved! uninstalled libraries continue to get installed on cluster startup

We have been trying to update some library versions by uninstalling the old versions and installing new ones. However, the old libraries continue to get installed on cluster startup despite not showing up in the "libraries" tab of the cluster page. W...

  • 2955 Views
  • 6 replies
  • 5 kudos
Latest Reply
reedzhang
New Contributor III
  • 5 kudos

The issue seemed to go away on its own. At some point the libraries page started showing what was getting installed to the cluster, and removing libraries from the page caused them to stop getting installed on cluster startup. I'm guessing there was ...

  • 5 kudos
5 More Replies
rachelk05
by New Contributor II
  • 1589 Views
  • 2 replies
  • 4 kudos

Resolved! Databricks Community: Cluster Terminated Reason: Unexpected Launch Failure

Hi,I've been encountering the following error when I try to start a cluster, but the status page says everything is fine. Is something happening or are there other steps I can try?Time2022-03-13 14:40:51 EDTMessageCluster terminated.Reason:Unexpected...

  • 1589 Views
  • 2 replies
  • 4 kudos
Latest Reply
Kaniz_Fatma
Community Manager
  • 4 kudos

Hi @Rachel Kelley​ , Were you able to not see the error anymore?

  • 4 kudos
1 More Replies
mikep
by New Contributor II
  • 3165 Views
  • 7 replies
  • 0 kudos

Resolved! Kubernetes or ZooKeeper for HA?

Hello. I am trying to understand High Availability in DataBricks. I understand that DB uses Kubernetes for the cluster manager and to manage Docker Containers. And while DB runs on top of AWS or Azure or GCP, is HA automatically provisioned when I st...

  • 3165 Views
  • 7 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

  • 0 kudos
6 More Replies
GoldenTuna
by New Contributor II
  • 3007 Views
  • 5 replies
  • 2 kudos

Resolved! Mounting an Azure Storage Account in a cluster init script?

We are trying to configure our environment so when our cluster starts up, it checks to see if we have mounted our Azure storage account container and if is not, mount it. We can do this fine in a notebook however have no luck doing this through an in...

  • 3007 Views
  • 5 replies
  • 2 kudos
Latest Reply
Anonymous
Not applicable
  • 2 kudos

@David Kruetzkamp​ - Would you be happy to mark whichever answer helped the most as best? That will help other members find the solution more quickly.

  • 2 kudos
4 More Replies
timothy_uk
by New Contributor III
  • 1919 Views
  • 4 replies
  • 0 kudos

Resolved! Zombie .Net Spark Databricks Job (CourseGrainedExecutorBackend)

Hi all,Environment:Nodes: Standard_E8s_v3Databricks Runtime: 9.0.NET for Apache Spark 2.0.0I'm invoking spark submit to run a .Net Spark job hosted in Azure Databricks. The job is written in C#.Net with its only transformation and action, reading a C...

  • 1919 Views
  • 4 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Moderator
  • 0 kudos

Hi @Timothy Lin​ ,I will recommend to not use spark.stop() or System.exit(0) in your code because it will explicitly stop the Spark context but the graceful shutdown and handshake with databricks' job service does not happen.

  • 0 kudos
3 More Replies
Mariusz_Cyp
by New Contributor II
  • 5287 Views
  • 3 replies
  • 7 kudos

When the billing time starts for the cluster?

Hi All, I'm just wondering when exactly the billing time starts for the DataBricks cluster? Is starting time included? If cluster creation time takes 3 minutes and query execution only 2, will I pay for 2 or 5?​Thanks in advance! MC

  • 5287 Views
  • 3 replies
  • 7 kudos
Latest Reply
franco_patano
New Contributor III
  • 7 kudos

Billing for databricks DBUs starts when Spark Context becomes available. Billing for the cloud provider starts when the request for compute is received and the VMs are starting up.

  • 7 kudos
2 More Replies
Hubert-Dudek
by Esteemed Contributor III
  • 745 Views
  • 0 replies
  • 19 kudos

docs.databricks.com

Databricks Runtime 10.2 Beta is available from yesterday.More details here: https://docs.databricks.com/release-notes/runtime/10.2.htmlNew features and improvementsUse Files in Repos with Spark StreamingDatabricks Utilities adds an update mount comma...

image.png
  • 745 Views
  • 0 replies
  • 19 kudos
Labels