Does anybody feel the unit test on Dataset is slow? (much slower than RDD). This is in Scala.

[tanin]

I profile it and it seems the slowness comes from Spark planning, especially for a more complex job (e.g. 100+ joins). Is there a way to speed it up (e.g. by disabling certain optimization)?

[Anonymous]

Hi @tanin​ 

Great to meet you, and thanks for your question! 

Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon.

Thanks.

[Noopur_Nigam]

Hi @tanin​ Are you writing unit testing code in Databricks notebook? If not, you may go through the below link:

https://docs.databricks.com/notebooks/testing.html

Please compare the result after writing your tests in notebook as well.

[tanin]

This is a unit test in Scala/Spark that is not in notebooks. It's in our repo.

[mcwir]

I had similar feeling recently.