https://community.databricks.com/t5/data-engineering/how-does-vectorized-pandas-udf-work/m-p/25288#M17569 <P><I>&gt;How does Vectorized Pandas UDF work?</I></P><P>Here is a video explaining the internals of Pandas UDFs (a.k.a. Vectorized UDFs) - <A href="https://youtu.be/UZl0pHG-2HA?t=123" target="test_blank">https://youtu.be/UZl0pHG-2HA?t=123</A> . They use <A href="http://arrow.apache.org/" alt="http://arrow.apache.org/" target="_blank">Apache Arrow</A>, to exchange data directly between JVM and Python driver/executors with near-zero (de)serialization cost.</P><P></P><P><I>&gt;Do Vectorized Pandas UDFs apply to batches of data sequentially or in parallel?</I></P><P>If let's say subtract_mean is a grouped map - when you run </P><PRE><CODE>df.groupby("id").apply(subtract_mean).show()</CODE></PRE><P>partitions in spark are converted into arrow record batches and depending on the cardinality of id, multiple batches would be processed in parallel. </P><P></P><P><I>&gt;And is there a way to set the batch size?</I></P><P>You could configure spark.sql.execution.arrow.maxRecordsPerBatch</P><P></P><P></P> Fri, 18 Jun 2021 00:23:35 GMT sajith_appukutt 2021-06-18T00:23:35Z https://community.databricks.com/t5/data-engineering/how-does-vectorized-pandas-udf-work/m-p/25287#M17568 <P>Do Vectorized Pandas UDFs apply to batches of data sequentially or in parallel? And is there a way to set the batch size?</P> Thu, 10 Jun 2021 17:57:58 GMT https://community.databricks.com/t5/data-engineering/how-does-vectorized-pandas-udf-work/m-p/25287#M17568 User16752246553 2021-06-10T17:57:58Z https://community.databricks.com/t5/data-engineering/how-does-vectorized-pandas-udf-work/m-p/25288#M17569 <P><I>&gt;How does Vectorized Pandas UDF work?</I></P><P>Here is a video explaining the internals of Pandas UDFs (a.k.a. Vectorized UDFs) - <A href="https://youtu.be/UZl0pHG-2HA?t=123" target="test_blank">https://youtu.be/UZl0pHG-2HA?t=123</A> . They use <A href="http://arrow.apache.org/" alt="http://arrow.apache.org/" target="_blank">Apache Arrow</A>, to exchange data directly between JVM and Python driver/executors with near-zero (de)serialization cost.</P><P></P><P><I>&gt;Do Vectorized Pandas UDFs apply to batches of data sequentially or in parallel?</I></P><P>If let's say subtract_mean is a grouped map - when you run </P><PRE><CODE>df.groupby("id").apply(subtract_mean).show()</CODE></PRE><P>partitions in spark are converted into arrow record batches and depending on the cardinality of id, multiple batches would be processed in parallel. </P><P></P><P><I>&gt;And is there a way to set the batch size?</I></P><P>You could configure spark.sql.execution.arrow.maxRecordsPerBatch</P><P></P><P></P> Fri, 18 Jun 2021 00:23:35 GMT https://community.databricks.com/t5/data-engineering/how-does-vectorized-pandas-udf-work/m-p/25288#M17569 sajith_appukutt 2021-06-18T00:23:35Z