cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta Streaming and Optimize

brickster_2018
Esteemed Contributor

I have a master delta table that is continuously getting written by a streaming job. I have optimize writes enabled and in addition, I run the OPTIMIZE command every 3 hours.

However, I think the downstream streaming jobs which are streaming the data from the master delta table do not get any benefit from the OPTIMIZE job. I tried stopping the downstream job and started it after completing the OPTIMIZE command, still, the downstream job is reading the un-optimized files

1 ACCEPTED SOLUTION

Accepted Solutions

brickster_2018
Esteemed Contributor

This is working as expected. For Delta streaming, the data files created in the first place will be used for streaming. The optimized files are not considered the downstream streaming job.

This is the reason it's not recommended to run VACUUM with fewer retention hours

View solution in original post

1 REPLY 1

brickster_2018
Esteemed Contributor

This is working as expected. For Delta streaming, the data files created in the first place will be used for streaming. The optimized files are not considered the downstream streaming job.

This is the reason it's not recommended to run VACUUM with fewer retention hours

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group