Hi!

We have a project where we do some Data Engineering for a client. I implemented a scheduled batch processing of Databricks' autoloader (stream w/ availableNow) since they primarily have numerous file exports from several sources. We wanted to follow the medallion architecture and ELT.

The endpoint of the to-be-built data warehouse, however, will be primarily a MySQL database hosted in Azure. As of now, my autoloader output goes to Delta tables but not in the default Hive metastore. Instead, I save them into a container in ADLS storage accounts sorted by source.

From here, I intend to build out the rest of the silver and gold layer tables in separate containers.

1. Based on my current approach and the intended use case, what barriers/challenges would I face? What would be good alternatives?
2. What is the actual best practice for handling Delta tables and the locations from where they are stored?

Appreciate any inputs provided!