What is the Best Practice of Maintaining the Delta table loaded in Streaming?

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

Hi Team,

We have our Bronze(append) Silver(append) and Gold(merge) Tables loaded using spark streaming continuously with trigger as processing time(3 secs).

We Also Run our Maintenance Job on the Table like OPTIMIZE,VACCUM and we perform DELETE for some tables with a datetime retention policy.

In such cases we see that our jobs often fails stating the underlying source file for deleted, or missing or updated...

I want to understand what is the optimized design or approach for my streaming process to perform this kind of Maintenance without affecting my streaming.

Thanks,

Naveen