Delta Live Tables use case

[Floody]

Hi all,

We have the following use case and wondering if DLT is the correct approach.

Landing area with daily dumps of parquet files into our Data Lake container.

The daily dump does a full overwrite of the parquet each time, keeping the same file name.

The idea would be to re-process the whole parquet each time and manage the delta in the bronze table with SCD 2.

Suggestions on the best approach would be helpful.

Cheers

[Kaniz]

Hi @Floody, Let’s explore how Delta Lake (DLT) can be a suitable approach for your use case.

1. Delta Lake Overview:

   • Delta Lake is an open source storage layer that brings ACID transactions to Apache Spark™ and big data workloads. It provides reliability, performance, and scalability for data lakes.
   • It’s built on top of existing data lakes and is compatible with Parquet, ORC, and other data formats.
   • Key features include ACID transactions, schema enforcement, time travel, and data versioning.

2. Your Use Case:

   • You have a daily dump of Parquet files into your data lake container.
   • The daily dump overwrites the Parquet files each time, but the file names remain the same.

3. Approaches to Consider:

   • Let’s discuss a few approaches using Delta Lake:

     a. Migrate Parquet Data with CLONE Parquet:

     • Incremental: Yes (supports additional data appended after conversion begins).
     • Duplicates Data: Yes (creates pointers to existing Parquet files).
     • Maintains Data Structure: Yes (maintains partitioning strategy).
     • Backfill Data: Yes (supports backfilling new files).
     • Ease of Use: Easy.
     • Details: Shallow clones create pointers to existing Parquet files, while deep clones copy all data files to a new location while converting to Delta Lake. Deep clones allow incremental detection of new files, including backfill operations¹.

     b. CONVERT TO DELTA:

     • Incremental: No.
     • Duplicates Data: No (converts data in place).
     • Maintains Data Structure: Yes.
     • Backfill Data: No.
     • Ease of Use: Easy.
     • Details: Converts existing Parquet data to Delta Lake format without incremental support¹.

     c. Auto Loader:

     • Incremental: Yes.
     • Duplicates Data: No.
     • Maintains Data Structure: No (optional).
     • Backfill Data: Some configuration required.
     • Ease of Use: Custom logic.
     • Details: Supports incremental loading from new files. Requires custom configuration¹.

     d. Custom Batch Spark Job:

     • Incremental: Yes (custom logic).
     • Duplicates Data: No.
     • Maintains Data Structure: No.
     • Backfill Data: Custom logic.
     • Ease of Use: Custom logic.
     • Details: Write a custom Spark job to handle incremental data and backfill operations¹.

4. Recommendation:

   • Considering your scenario, CLONE Parquet or CONVERT TO DELTA could be suitable.
   • If you need incremental support and want to maintain the existing partitioning structure, go for CLONE Parquet.
   • If you prefer a straightforward conversion without incremental support, choose CONVERT TO DELTA.

Remember to adapt the approach based on your specific requirements and workload characteristics. Delta Lake provides flexibility and robustness for managing data in your data lake container. 🌟🔗

For more detailed implementation steps, refer to the Azure Databricks documentation¹.