Converting from RDD to Dataset, and unit test takes 3x slower. (but prod is faster)

tanin — Sun, 06 Feb 2022 08:49:08 GMT

I converted a data job fro RDD to Dataset, and I've found that, in prod, the data job runs faster, which is nice.

But unit test runs 3x slower than before.

My best guess is that Dataset spends time doing a lot of stuffs like encoding, optimizing, querying planning.

I wonder if there's a way to disable most of this in unit test since all of these don't matter in a local test.

Alternatively, is there a way to cache the plan, so it doesn't need to be recomputed on each test?

Thank you!

topic Converting from RDD to Dataset, and unit test takes 3x slower. (but prod is faster) in Data Engineering