One thing I’d add is to think about governance of the derived data, not just access to the PROD source.If an analyst can read a governed PROD table and materialize the result into analytics_sandbox, that new table can now contain customer or PII data...
Nice starting point. One thing I’d change is not filtering out short-lived job clusters.Those clusters may be temporary, but the spend is real. I’d keep all billing records and map them back to something more stable: job, pipeline, application, team,...
I’d still keep a small retry policy, but I wouldn’t rely on retries as the concurrency strategy.If row-level concurrency is enabled and the jobs truly update different rows, those writes should normally be able to proceed independently. Retries are s...