I am in the process of designing a Medallion architecture where the data sources include REST API calls, JSON files, SQL Server, and Azure Event Hubs.
For the Silver and Gold layers, I plan to leverage Delta Live Tables (DLT). However, I am seeking guidance on the most effective approach to implement the Bronze layer, particularly in combination with Autoloader.
Specifically, for JSON file ingestion, I intend to use Autoloader with the trigger(availableNow=True) option. My understanding is that this option is not currently supported within DLT pipelines.
Could you please advise on recommended practices for implementing the Bronze layer to handle both batch and streaming ingestion scenarios in DLT, while ensuring compatibility with Autoloader?
