Hey everyone,
First time posting looking to complete my research for a Databricks implementation.
My current data pipeline consists of Azure Data Factory Pipelines that load data from ADLS into a SQL database. This happens once a day on schedule since the source team provides data to our ADLS. Decryption, processing, cleansing, encryption, file moving is all handled through notebooks that sink parquet files in ADLS and ultimately copy activities load to a SQL database. All of this is very procedural, sequential and quite frankly, digestible and straightforward when you see the flows and read the code.
I'm having trouble wrapping my head around the declarative nature of Spark Declarative Pipelines and why would I want to use Autoloader when it limits the processing power in between reading files and writing to tables.
I'll list some of my doubts and any insight on any of them is more than welcome.
- Is using autoloader's main advantage, reducing the overhead of manual file discovery and checkpointing for reads and writes?
- Can I still handle file decryption if I'm using autoloader?
- I've tested the writeStream for my sink to tables in Unity Catalog. Does the trigger AvailableNow effectively convert the stream into a batch load, suitable for my daily data intake?
- Is ForEachBatch the only way to handle procedural code in both notebooks using autoloader and in SDPs? What I mean by this is performing checks on a spark dataframe built from each file to be ingested, between the readStream and the writeStream when using Autoloader, to reject rows or files that do not comply with custom rules. In SDPs, performing updates to log tables seem to be only viable via forEachBatch.
I guess in summary I want to understand if what I need is a paradigm shift on the data processing for our Databricks implementation, leaning towards the declarative nature for table processing and handling file processing procedurally.
