https://community.databricks.com/t5/get-started-discussions/pandas-udod-max-batch-size-not-working-in-notebook/m-p/61914#M2760 <P>Hello&nbsp;</P><P>I am trying to set max batch size for pandas-udf in Databricks notebook, but in my tests it doesn’t have any effect on size.&nbsp;<SPAN class="">spark.conf.set("spark.sql.execution.arrow.enabled", "true")</SPAN></P><P><SPAN class="">spark.conf.set('spark.sql.execution.arrow.maxRecordsPerBatch', 1000000)</SPAN></P><P><SPAN class=""><BR /><SPAN>should I set it in other way like during cluster configuration?</SPAN></SPAN></P><P>Thanks for clarification in advance!</P><P>&nbsp;</P> Sat, 24 Feb 2024 18:20:57 GMT 277745 2024-02-24T18:20:57Z https://community.databricks.com/t5/get-started-discussions/pandas-udod-max-batch-size-not-working-in-notebook/m-p/61914#M2760 <P>Hello&nbsp;</P><P>I am trying to set max batch size for pandas-udf in Databricks notebook, but in my tests it doesn’t have any effect on size.&nbsp;<SPAN class="">spark.conf.set("spark.sql.execution.arrow.enabled", "true")</SPAN></P><P><SPAN class="">spark.conf.set('spark.sql.execution.arrow.maxRecordsPerBatch', 1000000)</SPAN></P><P><SPAN class=""><BR /><SPAN>should I set it in other way like during cluster configuration?</SPAN></SPAN></P><P>Thanks for clarification in advance!</P><P>&nbsp;</P> Sat, 24 Feb 2024 18:20:57 GMT https://community.databricks.com/t5/get-started-discussions/pandas-udod-max-batch-size-not-working-in-notebook/m-p/61914#M2760 277745 2024-02-24T18:20:57Z