- 10296 Views
- 5 replies
- 1 kudos
Getting non-traceable NullPointerExceptions
We're running a job that's issuing NullPointerException without traces of our job's code.Does anybody know what would be the best course of action when it comes to debugging these issues?The job is a Scala job running on DBR 11.3 LTS.In case it's rel...
- 10296 Views
- 5 replies
- 1 kudos
- 1 kudos
You could try enabling full stack traces and checking the Spark executor logs for hidden errors. Null Pointer Exceptions in Scala on DBR often come from lazy evaluations or missing schema fields during I/O. Reviewing your Data Frame transformations a...
- 1 kudos
- 4198 Views
- 5 replies
- 0 kudos
Resolved! Not received my certificate after passing Databricks Certified Associate Developer for Apache Spark 3.0 - Python.
I've successfully passed Databricks Certified Associate Developer for Apache Spark 3.0 - Python but still have not received the certificate. E-mail : anwarpatel91@gmail.com
- 4198 Views
- 5 replies
- 0 kudos
- 0 kudos
Even i am facing the same issue. I have successfully passed Databricks Certified Associate Developer for Apache Spark - Python but still have not received the certificate. E-mail : simha6.reddy@gmail.com
- 0 kudos
- 1097 Views
- 1 replies
- 1 kudos
I attended the Advanced Machine Learning with Databricks training last week virtually I am still unable to get the day 2 session videos of any of the...
I attended the Advanced Machine Learning with Databricks training last week virtually I am still unable to get the day 2 session videos of any of the Instructor led Paid Trainings. They are supposed to be available for replay with in 24 hours but I ...
- 1097 Views
- 1 replies
- 1 kudos
- 1 kudos
I have the same problem.
- 1 kudos
- 10654 Views
- 5 replies
- 5 kudos
Spark issue handling data from json when the schema DataType mismatch occurs
Hi,I have encountered a problem using spark, when creating a dataframe from a raw json source.I have defined an schema for my data and the problem is that when there is a mismatch between one of the column values and its defined schema, spark not onl...
- 10654 Views
- 5 replies
- 5 kudos
- 5 kudos
@Farzad Bonabi :Thank you for reporting this issue. It seems to be a known bug in Spark when dealing with malformed decimal values. When a decimal value in the input JSON data is not parseable by Spark, it sets not only that column to null but also ...
- 5 kudos
- 7084 Views
- 4 replies
- 2 kudos
Resolved! Databricks Spark Vs Spark on Yarn
I am moving my Spark workloads from EMR/on-premise Spark cluster to Databricks. I understand Databricks Spark is different from Yarn. How is the Databricks architecture different from yarn?
- 7084 Views
- 4 replies
- 2 kudos
- 2 kudos
What about the disadvantages?How can I separate multiple jobs running on the same cluster cleanly in the logs and same in the spark-ui?
- 2 kudos
- 21799 Views
- 6 replies
- 2 kudos
Resolved! MergeSchema Not Working
Hi All, I have a scenario where my Exisiting Delta Table looks like below:Now I have an incremental data with an additional column i.e. owner:Dataframe Name --> scdDFBelow is the code snippet to merge Incremental Dataframe to targetTable, but the new...
- 21799 Views
- 6 replies
- 2 kudos
- 2 kudos
In Databricks Runtime 15.2 and above, you can specify schema evolution in a merge statement using SQL or Delta table APIs:MERGE WITH SCHEMA EVOLUTION INTO targetUSING sourceON source.key = target.keyWHEN MATCHED THENUPDATE SET *WHEN NOT MATCHED THENI...
- 2 kudos
- 4022 Views
- 1 replies
- 2 kudos
6.4 Extended Support (includes Apache Spark 2.4.5, Scala 2.11 Connect Timeout
"Notebook detached Exception when creating execution context: java.net.SocketTimeout Exception: Connect Timeout" when trying to connect my cluster to a notebook. Then "Error trying to handle that request We failed to handle that request, please try a...
- 4022 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello @Retired_mod I am facing same issue I tried changing DBR but it is still giving me error and the cluster is not startingRegardsMS
- 2 kudos
- 15076 Views
- 3 replies
- 6 kudos
Resolved! How to add I custom logging in Databricks
I want to add custom logs that redirect in the Spark driver logs. Can I use the existing logger classes to have my application logs or progress message in the Spark driver logs.
- 15076 Views
- 3 replies
- 6 kudos
- 6 kudos
1) Is it possible to save all the custom logging to its own file? Currently it is being logging with all other cluster logs (see image) 2) Also Databricks it seems like a lot of blank files are also being created for this. Is this a bug? this include...
- 6 kudos
- 5606 Views
- 9 replies
- 3 kudos
Databricks Certified Associate Developer for Apache Spark 3.0
Databricks Certified Associate Developer for Apache Spark 3.0
- 5606 Views
- 9 replies
- 3 kudos
- 3 kudos
Hey I am looking for sample papers for the above exam other than the one provided by databricks do any one have any idea about it
- 3 kudos
- 4045 Views
- 2 replies
- 0 kudos
Resolved! The driver is temporarily unavailable
My job fails with Driver is temporarily unavailable. Apparently, it's permanently unavailable, because the job is not pausing but failing.
- 4045 Views
- 2 replies
- 0 kudos
- 0 kudos
I am facing the same issues . I am writing in batches using a simple for loop. I don't have any collect statements inside the loop. I am rewriting the partitions with partition overwrite dynamic mode in a huge wide delta table - several tb. The incr...
- 0 kudos
- 9283 Views
- 10 replies
- 9 kudos
Resolved! Request for reattempt voucher. Databricks Certified Associate Developer for Apache Spark 3.0 exam
Hi,I gave Databricks Certified Associate Developer for Apache Spark 3.0 exam today but missed by one percent. I got 68.33% and pass is 70%.I am planning to reattempt the exam, could you kindly give me another opportunity and provide reattempt voucher...
- 9283 Views
- 10 replies
- 9 kudos
- 9 kudos
Hi,I gave Databricks Certified Associate Developer for Apache Spark 3.0 Python exam yesterday but missed by three percent. I got 66.66% and pass is 70%.I am planning to reattempt the exam, could you kindly give me another opportunity and provide reat...
- 9 kudos
- 2879 Views
- 3 replies
- 2 kudos
KB Feedback Discussion In addition to the Databricks Community, we have a Support team that maintains a Knowledge Base (KB). The KB contains answers t...
KB Feedback DiscussionIn addition to the Databricks Community, we have a Support team that maintains a Knowledge Base (KB). The KB contains answers to common questions about Databricks, as well as information on optimisation and troubleshooting.These...
- 2879 Views
- 3 replies
- 2 kudos
- 6660 Views
- 5 replies
- 3 kudos
Resolved! Issue with Apache Spark™ Programming with Databricks course
Hello,I found an issue with the Apache Spark™ Programming with Databricks courses on Databricks Academy when trying to do the labs. The mount that the courses use for training data is failing with what looks to me like an authentication issue (see sc...
- 6660 Views
- 5 replies
- 3 kudos
- 3 kudos
I found the course Git Repo at (https://github.com/databricks-academy/apache-spark-programming-with-databricks-english), this works so using that instead of the 'apache-spark-programming-with-databricks.dbc' file available in the learning portal. #DA...
- 3 kudos
- 1199 Views
- 0 replies
- 1 kudos
Missed Associate Developer for Apache Spark 3.0 - Python Due to Power outage
Dear Databricks Certification Team,Unfortunately, I was unable to take the exam as scheduled due to an unforeseen power breakdown in my area. The power outage occurred just before the exam, rendering me unable to access the necessary resources to com...
- 1199 Views
- 0 replies
- 1 kudos
- 3399 Views
- 3 replies
- 3 kudos
Databricks run-time 12.2 LTS drop function problem
I am getting an error about the `drop function of pyspark` at a cluster using 12.2 LTS. When I check the error I see spark solved that bug, see SPARK-42444. Also when I check maintenance updates page, I saw this solved issue included the Databricks R...
- 3399 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Sevval Mehder Elevate our community by acknowledging exceptional contributions. Your participation in marking the best answers is a testament to our collective pursuit of knowledge.
- 3 kudos