- 2762 Views
- 2 replies
- 0 kudos
Slow running Spark job issue - due to the unknown spark stages created by Databircks Compute cluster
Hi Team,Recently we migrated the spark jobs from self hosted spark(YARN) Cluster to Databricks.Currently we are using the Databricks workflows with Job_Compute clusters and the Job Type - Spark JAR type execution, so when we run the job in databricks...
- 2762 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey Brahma,Thanks for your reply. As a first step I will disable AQE config and test it. We are using the node pools with job_compute cluster type so that its not spinning up a new cluster for each Job. I'm configuring the below two configs also, do ...
- 0 kudos
- 1773 Views
- 1 replies
- 0 kudos
Legacy Autoscaling(workflow) VS Enhanced Autoscaling(DLT)
I conducted a proof of concept (POC) to compare the performance of the DLT pipeline and Databricks Workflow using the same workload, task, code, and cluster configuration. Both configurations were set with autoscaling enabled, with a minimum of 1 wor...
- 1773 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi chsoni12,How are you doing today?, As per my understanding, That's a great observation, and it's awesome that you're testing performance and cost between DLT and regular workflows. The key difference here lies in how autoscaling works. DLT pipelin...
- 0 kudos
- 3964 Views
- 4 replies
- 0 kudos
i can list out the file using dbutils but can not able to read files in databricks
I can list out the file using dbutils but can not able to read files in databricks. PFB in screenshot. I can able to see the file using dbutils.fs.ls but when i try to read this file using read_excel then it is showing me an error like "FileNotFound...
- 3964 Views
- 4 replies
- 0 kudos
- 0 kudos
Hello @MohammadWasi did you finally figure out what the problem was? I am encountering the exact same issue
- 0 kudos
- 994 Views
- 1 replies
- 0 kudos
Folders in Workflows/Jobs
Would be great if we could "group" Workflows/Jobs in Databricks using folders.This way, the Workflows list won't be too cluttered with all Workflows/Jobs in the same root-level.
- 994 Views
- 1 replies
- 0 kudos
- 0 kudos
Hello @BMex! You can submit this as a feature request through the Databricks Ideas Portal. This helps the product team consider it for future improvements
- 0 kudos
- 2280 Views
- 5 replies
- 1 kudos
Resolved! Fails to use unity catalog in All purpose cluster
Hey there,today we cannot load/read data from Unity Catalog with the same cluster as we did yesterday successfully (no changes in clsuter configuration).The error, which persists, according to the cluster logs is:com.databricks.common.client.Databric...
- 2280 Views
- 5 replies
- 1 kudos
- 1 kudos
@Advika the issue is gone.Now without any change all-purpose has access again to unity catalog.Who knows what happened...Thanks again for your interest
- 1 kudos
- 3483 Views
- 2 replies
- 1 kudos
Resolved! Intermittent Timeout Error While Waiting for Python REPL to Start in Databricks
Hi everyone,I’ve been encountering an error that says "Timeout while waiting for the Python REPL to start. Took longer than 60 seconds" during my work in Databricks. The issue seems to happen intermittently - sometimes the REPL starts without any pro...
- 3483 Views
- 2 replies
- 1 kudos
- 1 kudos
@Rohan2405"If everything else is in place, increasing the REPL startup timeout in the cluster configuration may help accommodate slower setups".Can you please guide me how to increase the REPL timeout in cluster configuration? Like I've add this conf...
- 1 kudos
- 4923 Views
- 2 replies
- 2 kudos
Spark Driver keeps restarting due to high GC pressure despite scaling up memory
I'm running into an issue where my Spark driver keeps pausing and eventually restarting due to excessive garbage collection (GC), even though I’ve already scaled up the cluster memory. Below is an example from the driver logs:Driver/192.168.231.23 pa...
- 4923 Views
- 2 replies
- 2 kudos
- 2 kudos
Thank you very much for your detailed analysis and helpful recommendations.We have reviewed your suggestions, and I’d like to share a quick update:We have already tried most of the mitigation strategies you mentioned — including increasing driver mem...
- 2 kudos
- 2648 Views
- 1 replies
- 0 kudos
Policy for DLT
Hi,I am trying to define a policy for our DLT pipelines and I would like to provide a specific spark version like in the below example: { "spark_conf.spark.databricks.cluster.profile": { "type": "forbidden", "hidden": true }, "spark_ve...
- 2648 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @ankit001mittal The error you're encountering is because Delta Live Tables (DLT) has specific requirements and automatically manages certain cluster configurations, including the Spark version. DLT pipelines are designed to use optimized Spark ver...
- 0 kudos
- 11956 Views
- 4 replies
- 1 kudos
Dashboard always display truncated data
Hello, we're working with a serverless SQL cluster to query Delta tables and display some analytics in dashboards. We have some basic group by queries that generate around 36k lines, and they are executed without the "limit" key word. So in the data ...
- 11956 Views
- 4 replies
- 1 kudos
- 1 kudos
I also have this issue, 95% of our reporting goes to excel from display window.. We need the full data shown so can simply copy and paste to excel , no need to "Download", this causes unneeded files piled up in the download folder, it also as you s...
- 1 kudos
- 3773 Views
- 2 replies
- 0 kudos
Resolved! Connection Databricks Postgresql
I use Databricks and I try to connect to posgresql via the following code"jdbcHostname = "xxxxxxx"jdbcDatabase = "xxxxxxxxxxxx"jdbcPort = "5432"username = "xxxxxxx"password = "xxxxxxxx"jdbcUrl = "jdbc:postgresql://{0}:{1}/{2}".format(jdbcHostname, jd...
- 3773 Views
- 2 replies
- 0 kudos
- 0 kudos
Can you tell me how you are able to connect postgres database from Databricks . Do we have to whitelist ips in postgres?
- 0 kudos
- 3974 Views
- 4 replies
- 2 kudos
Resolved! Databricks Connect and DBR 16.4 LTS @ Scala 2.13
Hi there,we're running Scala jobs on Databricks and I was eager to finally upgrade to Scala 2.13. However, Databricks Connect 16.4.x doesn't handle Scala versioning, so all dependencies are tied to Scala 2.13. It's rather tedious to exclude all 2.12 ...
- 3974 Views
- 4 replies
- 2 kudos
- 2 kudos
Runtime 17.0 is out in beta right now and I expect it to GA in the near future. Keep an eye out for the runtime release notes. Once they are released you will be able to see what's present and hopefully (fingers crossed) your dependency issue will be...
- 2 kudos
- 3943 Views
- 3 replies
- 3 kudos
Resolved! Understanding Serverless Compute Sharing Across Notebooks in Databricks
Hi Community,I am using Databricks Serverless compute in notebooks. When I create multiple notebooks and choose Serverless as the compute, I noticed that I can select the same serverless cluster for all of them.This brings up a few questions:Is this ...
- 3943 Views
- 3 replies
- 3 kudos
- 3 kudos
Could you clarify what you mean by “The driver detaches”? If the driver detaches, the cluster would typically fail. Are you using Spark for processing, or is this a pure Python workload? If you’re using pure Python, only the driver node is utilized, ...
- 3 kudos
- 1754 Views
- 2 replies
- 2 kudos
Resolved! azure-storage-blob-changefeed -ModuleNotFoundError: No module named 'azure.storage.blob.changefeed'
Added a python package azure-storage-blob-changefeed (https://pypi.org/project/azure-storage-blob-changefeed/).But trying to access it from the notebook like this:from azure.storage.blob.changefeed import ChangeFeedClientraises ModuleNotFoundError.Tr...
- 1754 Views
- 2 replies
- 2 kudos
- 2 kudos
Hi! thanks for answering to our question.We tried that and it didn't work; but it gave us the idea to uninstall and install the last version azure-storage-blob, it worked after that.For serverless clusters, the package: azure-storage-blob is coming w...
- 2 kudos
- 9985 Views
- 5 replies
- 0 kudos
Encountering Error UNITY_CREDENTIAL_SCOPE_MISSING_SCOPE
As of this morning we started receiving the following error message on a Databricks job with a single Pyspark Notebook task. The job has not had any code changes in 2 months. The cluster configuration has also not changed. The last successful run of ...
- 9985 Views
- 5 replies
- 0 kudos
- 0 kudos
Im facing the same issue. The error came after we switched to unity catalog, and I try to replay a stream job. The job both reads and writes. This problem has been open for more than 12 months and only new contributes has commented on it...The stackt...
- 0 kudos
- 3813 Views
- 4 replies
- 1 kudos
Resolved! Data Engineer Associate Certificate and badge not yet received
@Kaniz Fatma I have cleared the certification exam on 26th January 2023, but still haven't received the certificate. I had given the exam with a different mail ID but I'm not receiving any emails from Databricks to that mail ID.Kindly help me resolv...
- 3813 Views
- 4 replies
- 1 kudos
- 1 kudos
Hi @Naeem_K I am also facing same issue can you please tell me how to resolve this
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
