cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Does anybody feel the unit test on Dataset is slow? (much slower than RDD). This is in Scala.

tanin
Contributor

I profile it and it seems the slowness comes from Spark planning, especially for a more complex job (e.g. 100+ joins). Is there a way to speed it up (e.g. by disabling certain optimization)?

4 REPLIES 4

Anonymous
Not applicable

Hi @tanin​ 

Great to meet you, and thanks for your question! 

Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon.

Thanks.

Noopur_Nigam
Valued Contributor II
Valued Contributor II

Hi @tanin​ Are you writing unit testing code in Databricks notebook? If not, you may go through the below link:

https://docs.databricks.com/notebooks/testing.html

Please compare the result after writing your tests in notebook as well.

This is a unit test in Scala/Spark that is not in notebooks. It's in our repo.

mcwir
Contributor

I had similar feeling recently.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!