Ryan_Chynoweth
Databricks Employee
Databricks Employee

Check out our auto loader capabilities that can automatically track and process files that need to be processed.

Autoloader

There are two options:

  • directory listing, which is essentially completing the same steps that you have listed above but in a slightly more efficient manner.
  • file notification, which creates managed resources in order to track files using a Azure Event Grid and Queue Storage services.

The file notification option is more scalable and is likely to better suit your needs.