@Kesavan31 , you’re calling out a problem a lot of teams run into in practice—usually only after something breaks and nobody’s quite sure why.
What really stands out to me is that you didn’t try to fight the platform. You leaned into the reality that file arrival triggers are Create-event driven and designed a clean, deterministic pattern around that constraint. The flag file mechanism is simple, intentional, and—most importantly—production-proven.
It also avoids all the usual workarounds we’ve all seen: timestamped filenames, forced renames, cron jobs, or bolting on an external scheduler just to paper over storage behavior. Instead, you establish a clear contract between CI/CD, ADLS, and Databricks that guarantees a trigger when it actually matters.
That delete-and-reintroduce loop is the fundamental insight. It turns a platform limitation into a predictable automation pattern and restores something easy to lose in data platforms: trust. When metadata changes, pipelines run. Every time.
Sometimes the best solutions aren’t complex—they’re thoughtfully engineered. This is an excellent example of that.
