Hi @fjrodriguez ,

What about using databricks autoloader and triggering workflow every 15 min? Autoloader automatically detects what new files has arrived since last trigger of a job and will load only new files to target table. You can use available now trigger option which consumes all available records as an incremental batch.

So, let's say you prepare a notebook that will use autoloader. Now you will schedule this notebook using databricks workflows with option Max concurrent runs = 1. This will ensure that your job will run every 15 minutes, it will consume all new files that appeared within that period and if processing takes longer than 15 minutes it will wait for a previois job to finish,