topic Calling a python function (def) in databricks in Data Engineering

Calling a python function (def) in databricks

pjp94 — Mon, 05 Dec 2022 20:53:47 GMT

Not sure if I'm missing something here, but running a task outside of a python function runs much much quicker than executing the same task inside a function. Is there something I'm missing with how spark handles functions?

def task(x):

y = dostuff(x)

return y

y = dostuff(x)

Re: Calling a python function (def) in databricks

Bartek — Mon, 05 Dec 2022 23:32:25 GMT

Hi @pjp, could you provide some more information? I'm not aware of any mechanism in Spark that can have such impact, but maybe example will make it easier for community to replicate, perform some benchmarking and help you.

Cheers

Bartek

Re: Calling a python function (def) in databricks

pjp94 — Mon, 05 Dec 2022 23:49:19 GMT

Sure. My function queries an external database (jdbc) along with a delta table. I'm not performing any expensive computations - just filtering for the most part. When printing timestamps in the function, I notice that most of the time is being spent on the latter (delta table query/manipulations). I don't know why that is. I even cache the tables when I query. When I functionalize, the result takes 15 min and if I run outside of a function, it takes 3 min.

Re: Calling a python function (def) in databricks

Ajay-Pandey — Tue, 06 Dec 2022 09:22:21 GMT

UDF is more expensive in Spark

That could be the reason for this

Re: Calling a python function (def) in databricks

Bartek — Tue, 06 Dec 2022 10:04:31 GMT

yes, there is difference in performance between Python and Scala - still, @Paras Patel sees performance penalty using Python in both cases

Re: Calling a python function (def) in databricks

Hubert-Dudek — Tue, 06 Dec 2022 11:06:48 GMT

It would be easier if you share whole your code @pjp94

Re: Calling a python function (def) in databricks

UmaMahesh1 — Tue, 06 Dec 2022 12:40:11 GMT

Assuming that dostuff you mentioned is a spark sql function, you can take a look at this stack overflow thread and links in the same thread to get some idea.

https://stackoverflow.com/questions/38296609/spark-functions-vs-udf-performance

Re: Calling a python function (def) in databricks

huyd — Thu, 29 Dec 2022 22:51:17 GMT

If you can convert your Python udfs to sql udfs. These play nice adaptive query executions and won’t have performance penalties.

Re: Calling a python function (def) in databricks

ramravi — Mon, 02 Jan 2023 14:07:19 GMT

Seems to be you are using UDF here. UDFs in spark are expensive because spark doesn't know how to optimize the UDF. Better to avoid them unless you have no other choice.

Re: Calling a python function (def) in databricks

sher — Fri, 06 Jan 2023 06:30:06 GMT

don't use python normal function use UDF in pyspark so that will be faster