- 11545 Views
- 3 replies
- 0 kudos
- 11545 Views
- 3 replies
- 0 kudos
- 0 kudos
I've also run into the same issue, customised docker image does not give DATABRICKS_RUNTIME_VERSION as env. I believe there are still many issues in how customised docker image is used in databricks cluster.Can anyone from databricks help answer it?
- 0 kudos
- 2907 Views
- 5 replies
- 4 kudos
Resolved! Out of Memory/Connection Lost When Writing to External SQL Server from Databricks Using JDBC Connection
I am working on writing a large amount of data from Databricks to an external SQL server using a JDB connection. I keep getting timeout errors/connection lost but digging deeper it appears to be a memory problem. I am wondering what cluster configura...
- 2907 Views
- 5 replies
- 4 kudos
- 3591 Views
- 4 replies
- 3 kudos
Resolved! Single-Node cluster works but Multi-Node clusters do not read data.
I am currently working with a VNET injected databricks workspace. At the moment I have mounted a the databricks cluster on an ADLS G2 resource. When running notebooks on a single node that read, transform, and write data we do not encounter any probl...
- 3591 Views
- 4 replies
- 3 kudos
- 3 kudos
@TheDataDexter Did you find a solution to your problem? I am facing the same issue
- 3 kudos
- 22302 Views
- 8 replies
- 9 kudos
Is there any way to monitor the CPU, disk and memory usage of a cluster while a job is running?
I am looking for something preferably similar to Windows task manager which we can use for monitoring the CPU, memory and disk usage for local desktop.
- 22302 Views
- 8 replies
- 9 kudos
- 9 kudos
Some important info to look in Gangalia UI in CPU, memory and server load charts to spot the problem:CPU chart :User %Idle %High percentage of user % indicates heavy CPU usage in the cluster.Memory chart : Use %Free %Swap % If you see purple line ove...
- 9 kudos
- 4933 Views
- 5 replies
- 2 kudos
Resolved! New spark cluster being configured in local mode
Hi,We have two workspaces on Databricks, prod and dev. On prod, if we create a new all-purpose cluster through the web interface and go to Environment in the the spark UI, the spark.master setting is correctly set to be the host IP. This results in a...
- 4933 Views
- 5 replies
- 2 kudos
- 2 kudos
I found the same issue when choosing the default cluster setup on first setup that when I went to edit the cluster to add an instance profile, I was not able to save without fixing this. Thanks for the tip
- 2 kudos
- 9435 Views
- 7 replies
- 18 kudos
Resolved! Cluster Modes - High Concurrency
It took me quite some time to find the option to create a cluster in High Concurrency mode. It was hidden in the new UI.What should be the way to access the data with TAC?What is the equivalent mode to work with TAC ?Does it mean that we are being pu...
- 9435 Views
- 7 replies
- 18 kudos
- 4741 Views
- 3 replies
- 3 kudos
How do I move existing workflows and jobs running on an all-purpose cluster to a shared jobs cluster?
A Databricks cluster is a set of computation resources that performs the heavy lifting of all of the data workloads you run in Databricks. Databricks provides a number of options when you create and configure clusters to help you get the best perform...
- 4741 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Madelyn Mullen​ , Thank you for sharing such an excellent and informative post. We hope to see these very often.
- 3 kudos
- 5417 Views
- 13 replies
- 7 kudos
Resolved! Error saving changes on Job Cluster
Hello all and thanks.After apply to serving a model, I go to edit corresponding Job Cluster to configure its init_script but when I try to save changes (Confirm and restart) it thrown the following error:Error: Cannot edit cluster 0503-141315-hu3wd4i...
- 5417 Views
- 13 replies
- 7 kudos
- 7 kudos
Sorry for the delay in responding. Finally a partner could fix the problem, he can edit without problems the cluster and add the init_script.Thank you!
- 7 kudos
- 3620 Views
- 6 replies
- 5 kudos
Resolved! uninstalled libraries continue to get installed on cluster startup
We have been trying to update some library versions by uninstalling the old versions and installing new ones. However, the old libraries continue to get installed on cluster startup despite not showing up in the "libraries" tab of the cluster page. W...
- 3620 Views
- 6 replies
- 5 kudos
- 5 kudos
The issue seemed to go away on its own. At some point the libraries page started showing what was getting installed to the cluster, and removing libraries from the page caused them to stop getting installed on cluster startup. I'm guessing there was ...
- 5 kudos
- 1808 Views
- 2 replies
- 4 kudos
Resolved! Databricks Community: Cluster Terminated Reason: Unexpected Launch Failure
Hi,I've been encountering the following error when I try to start a cluster, but the status page says everything is fine. Is something happening or are there other steps I can try?Time2022-03-13 14:40:51 EDTMessageCluster terminated.Reason:Unexpected...
- 1808 Views
- 2 replies
- 4 kudos
- 3736 Views
- 7 replies
- 0 kudos
Resolved! Kubernetes or ZooKeeper for HA?
Hello. I am trying to understand High Availability in DataBricks. I understand that DB uses Kubernetes for the cluster manager and to manage Docker Containers. And while DB runs on top of AWS or Azure or GCP, is HA automatically provisioned when I st...
- 3736 Views
- 7 replies
- 0 kudos
- 3551 Views
- 5 replies
- 2 kudos
Resolved! Mounting an Azure Storage Account in a cluster init script?
We are trying to configure our environment so when our cluster starts up, it checks to see if we have mounted our Azure storage account container and if is not, mount it. We can do this fine in a notebook however have no luck doing this through an in...
- 3551 Views
- 5 replies
- 2 kudos
- 2 kudos
@David Kruetzkamp​ - Would you be happy to mark whichever answer helped the most as best? That will help other members find the solution more quickly.
- 2 kudos
- 2342 Views
- 4 replies
- 0 kudos
Resolved! Zombie .Net Spark Databricks Job (CourseGrainedExecutorBackend)
Hi all,Environment:Nodes: Standard_E8s_v3Databricks Runtime: 9.0.NET for Apache Spark 2.0.0I'm invoking spark submit to run a .Net Spark job hosted in Azure Databricks. The job is written in C#.Net with its only transformation and action, reading a C...
- 2342 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @Timothy Lin​ ,I will recommend to not use spark.stop() or System.exit(0) in your code because it will explicitly stop the Spark context but the graceful shutdown and handshake with databricks' job service does not happen.
- 0 kudos
- 6139 Views
- 3 replies
- 11 kudos
When the billing time starts for the cluster?
Hi All, I'm just wondering when exactly the billing time starts for the DataBricks cluster? Is starting time included? If cluster creation time takes 3 minutes and query execution only 2, will I pay for 2 or 5?​Thanks in advance! MC
- 6139 Views
- 3 replies
- 11 kudos
- 11 kudos
Billing for databricks DBUs starts when Spark Context becomes available. Billing for the cloud provider starts when the request for compute is received and the VMs are starting up.
- 11 kudos
- 855 Views
- 0 replies
- 19 kudos
docs.databricks.com
Databricks Runtime 10.2 Beta is available from yesterday.More details here: https://docs.databricks.com/release-notes/runtime/10.2.htmlNew features and improvementsUse Files in Repos with Spark StreamingDatabricks Utilities adds an update mount comma...
- 855 Views
- 0 replies
- 19 kudos