spark code optimization

[Yogic24]

𝐂𝐨𝐝𝐞 𝐎𝐩𝐭𝐢𝐦𝐢𝐳𝐚𝐭𝐢𝐨𝐧 𝐢𝐧 𝐏𝐲𝐒𝐩𝐚𝐫𝐤

Please refer attach document
While exploring, I discovered an excellent document by Deepa Vasanthkumar that effectively explains spark code optimization techniques.

Like and reshare ✅

[Witold · Accepted solution]

Sorry, but that's neither good, nor complete and actually also wrong (like the part about caching / persistence).

Rather than that I'd strongly advice that you spread Databricks official Comprehensive Guide to Optimize Databricks, Spark and Delta Lake Workloads 

[Deepa_data_ai]

First Reply 🙂 

Thank you @Yogic24 for sharing my blog post, This may not be complete guide, as it was meant to be only to include crisp info for a tiny blog. However any optimization would be based on the problem which we are trying to solve, and may be dependent on a case to case basis. 

In any case, official documentation/source code should be the first choice to refer for any optimizations. 

Thank you & Have a great day !

Deepa Vasanthkumar