https://community.databricks.com/t5/data-engineering/delta-streaming-and-optimize/m-p/18908#M12606 <P>I have a master delta table that is continuously getting written by a streaming job. I have optimize writes enabled and in addition, I run the OPTIMIZE command every 3 hours. </P><P>However, I think the downstream streaming jobs which are streaming the data from the master delta table do not get any benefit from the OPTIMIZE job. I tried stopping the downstream job and started it after completing the OPTIMIZE command, still, the downstream job is reading the un-optimized files </P> Fri, 25 Jun 2021 19:24:56 GMT brickster_2018 2021-06-25T19:24:56Z https://community.databricks.com/t5/data-engineering/delta-streaming-and-optimize/m-p/18908#M12606 <P>I have a master delta table that is continuously getting written by a streaming job. I have optimize writes enabled and in addition, I run the OPTIMIZE command every 3 hours. </P><P>However, I think the downstream streaming jobs which are streaming the data from the master delta table do not get any benefit from the OPTIMIZE job. I tried stopping the downstream job and started it after completing the OPTIMIZE command, still, the downstream job is reading the un-optimized files </P> Fri, 25 Jun 2021 19:24:56 GMT https://community.databricks.com/t5/data-engineering/delta-streaming-and-optimize/m-p/18908#M12606 brickster_2018 2021-06-25T19:24:56Z https://community.databricks.com/t5/data-engineering/delta-streaming-and-optimize/m-p/18909#M12607 <P>This is working as expected. For Delta streaming, the data files created in the first place will be used for streaming. The optimized files are not considered the downstream streaming job. </P><P></P><P>This is the reason it's not recommended to run VACUUM with fewer retention hours</P> Fri, 25 Jun 2021 19:26:24 GMT https://community.databricks.com/t5/data-engineering/delta-streaming-and-optimize/m-p/18909#M12607 brickster_2018 2021-06-25T19:26:24Z