https://community.databricks.com/t5/data-engineering/how-can-i-run-optimize-on-a-table-if-i-am-streaming-to-it-24-7/m-p/19594#M13151 <P>If the streaming job is making bling appends to the delta table, then it's perfectly fine to run OPTIMIZE query in parallel.</P><P>However, if the streaming job is performing MERGE or UPDATE then it can conflict with the OPTIMIZE operations. In such cases within the streaming, custom logic can be written to perform the optimize as part of the streaming job itself. Maybe every 100 batches perform the OPTIMIZE. </P><P>Check here for the list of operations:</P><P><A href="https://docs.databricks.com/delta/concurrency-control.html#write-conflicts" target="test_blank">https://docs.databricks.com/delta/concurrency-control.html#write-conflicts</A></P> Fri, 25 Jun 2021 21:28:20 GMT brickster_2018 2021-06-25T21:28:20Z https://community.databricks.com/t5/data-engineering/how-can-i-run-optimize-on-a-table-if-i-am-streaming-to-it-24-7/m-p/19593#M13150 <P>I have a table that I need to be continuously streaming into. I know it's best practice to run Optimize on my tables periodically. But if I never stop writing to the table, how and when can I run OPTIMIZE against it?</P> Fri, 25 Jun 2021 16:15:20 GMT https://community.databricks.com/t5/data-engineering/how-can-i-run-optimize-on-a-table-if-i-am-streaming-to-it-24-7/m-p/19593#M13150 User16826992666 2021-06-25T16:15:20Z https://community.databricks.com/t5/data-engineering/how-can-i-run-optimize-on-a-table-if-i-am-streaming-to-it-24-7/m-p/19594#M13151 <P>If the streaming job is making bling appends to the delta table, then it's perfectly fine to run OPTIMIZE query in parallel.</P><P>However, if the streaming job is performing MERGE or UPDATE then it can conflict with the OPTIMIZE operations. In such cases within the streaming, custom logic can be written to perform the optimize as part of the streaming job itself. Maybe every 100 batches perform the OPTIMIZE. </P><P>Check here for the list of operations:</P><P><A href="https://docs.databricks.com/delta/concurrency-control.html#write-conflicts" target="test_blank">https://docs.databricks.com/delta/concurrency-control.html#write-conflicts</A></P> Fri, 25 Jun 2021 21:28:20 GMT https://community.databricks.com/t5/data-engineering/how-can-i-run-optimize-on-a-table-if-i-am-streaming-to-it-24-7/m-p/19594#M13151 brickster_2018 2021-06-25T21:28:20Z