https://community.databricks.com/t5/data-engineering/while-trying-to-save-the-spark-dataframe-to-delta-table-is/m-p/4028#M870 <P>While working on video analytics task I need to save the image bytes to the delta table earlier extracted into the spark dataframe. While I want to over write a same delta table over the period of complete task and also the size of input data differs. It is taking too much time even after doing several trials with compactions. I cant use the streaming delta tables as I simply want to store extracted image bytes to the delta table and simply complete the inference task for object detection and other transformations. I have even tried to drop the lengthy data columns but did not make any difference. 1 Driver</P><P>16&nbsp;GB Memory,&nbsp;4&nbsp;Cores 11.3.x-gpu-ml-scala2.12, g4dn.xlarge is the configuration of my current cluster.</P><P>11.3.x-gpu-ml-scala2.12</P> Wed, 24 May 2023 12:08:10 GMT Neil 2023-05-24T12:08:10Z https://community.databricks.com/t5/data-engineering/while-trying-to-save-the-spark-dataframe-to-delta-table-is/m-p/4028#M870 <P>While working on video analytics task I need to save the image bytes to the delta table earlier extracted into the spark dataframe. While I want to over write a same delta table over the period of complete task and also the size of input data differs. It is taking too much time even after doing several trials with compactions. I cant use the streaming delta tables as I simply want to store extracted image bytes to the delta table and simply complete the inference task for object detection and other transformations. I have even tried to drop the lengthy data columns but did not make any difference. 1 Driver</P><P>16&nbsp;GB Memory,&nbsp;4&nbsp;Cores 11.3.x-gpu-ml-scala2.12, g4dn.xlarge is the configuration of my current cluster.</P><P>11.3.x-gpu-ml-scala2.12</P> Wed, 24 May 2023 12:08:10 GMT https://community.databricks.com/t5/data-engineering/while-trying-to-save-the-spark-dataframe-to-delta-table-is/m-p/4028#M870 Neil 2023-05-24T12:08:10Z https://community.databricks.com/t5/data-engineering/while-trying-to-save-the-spark-dataframe-to-delta-table-is/m-p/4029#M871 <P>can you check the spark UI, to see where the time is spent?</P><P>It can be a join, udf, ...</P> Thu, 25 May 2023 07:52:58 GMT https://community.databricks.com/t5/data-engineering/while-trying-to-save-the-spark-dataframe-to-delta-table-is/m-p/4029#M871 -werners- 2023-05-25T07:52:58Z