topic OOP programming in Pyspark on Databricks platform in Data Engineering

OOP programming in Pyspark on Databricks platform

AtanuC — Wed, 20 Sep 2023 05:30:48 GMT

Hello Expert,

I have a doubt so I need your advice and opinion on below query.

Does OOP is a good chioce of programming for distributed data processing ? like Pysaprk in Databricks platform ? If not then what it is and what kinfd of challenges could be there ? or will the functional programming approach be the best option in this case?

Please help me to understand the concept.

Thanks!!!

Re: OOP programming in Pyspark on Databricks platform

NandiniN — Sat, 01 Feb 2025 07:31:13 GMT

Functional programming is generally better suited for distributed data processing with PySpark on Databricks due to its emphasis on immutability, stateless operations, and higher-order functions. These features align well with Spark's execution model and make it easier to write scalable and maintainable code. While OOP can be used, it introduces additional complexity and challenges that are better avoided in a distributed environment.

https://docs.databricks.com/en/pyspark/index.html