topic Re: Does anybody feel the unit test on Dataset is slow? (much slower than RDD). This is in Scala. in Data Engineering

Does anybody feel the unit test on Dataset is slow? (much slower than RDD). This is in Scala.

tanin — Tue, 25 Oct 2022 19:42:18 GMT

I profile it and it seems the slowness comes from Spark planning, especially for a more complex job (e.g. 100+ joins). Is there a way to speed it up (e.g. by disabling certain optimization)?

Re: Does anybody feel the unit test on Dataset is slow? (much slower than RDD). This is in Scala.

Anonymous — Mon, 28 Nov 2022 04:12:33 GMT

Hi @tanin

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon.

Thanks.

Re: Does anybody feel the unit test on Dataset is slow? (much slower than RDD). This is in Scala.

Noopur_Nigam — Wed, 30 Nov 2022 15:30:59 GMT

Hi @tanin Are you writing unit testing code in Databricks notebook? If not, you may go through the below link:

https://docs.databricks.com/notebooks/testing.html

Please compare the result after writing your tests in notebook as well.

Re: Does anybody feel the unit test on Dataset is slow? (much slower than RDD). This is in Scala.

tanin — Wed, 30 Nov 2022 17:25:41 GMT

This is a unit test in Scala/Spark that is not in notebooks. It's in our repo.

Re: Does anybody feel the unit test on Dataset is slow? (much slower than RDD). This is in Scala.

mcwir — Thu, 01 Dec 2022 09:14:55 GMT

I had similar feeling recently.