cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Does anybody feel the unit test on Dataset is slow? (much slower than RDD). This is in Scala.

tanin
Contributor

I profile it and it seems the slowness comes from Spark planning, especially for a more complex job (e.g. 100+ joins). Is there a way to speed it up (e.g. by disabling certain optimization)?

4 REPLIES 4

Anonymous
Not applicable

Hi @taninโ€‹ 

Great to meet you, and thanks for your question! 

Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon.

Thanks.

Noopur_Nigam
Valued Contributor II

Hi @taninโ€‹ Are you writing unit testing code in Databricks notebook? If not, you may go through the below link:

https://docs.databricks.com/notebooks/testing.html

Please compare the result after writing your tests in notebook as well.

This is a unit test in Scala/Spark that is not in notebooks. It's in our repo.

mcwir
Contributor

I had similar feeling recently.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group