Streaming with Delta table source- definition of "File"?

Michael_Galli
Databricks Partner

‎07-04-2022 02:11 AM

Hi all,

I have a Delta Table as a Spark Streaming source.

This table contains signals on row level -> each signal is one append to the source table that creates a new version in the delta transaction history.

I am not really sure now how Spark streaming works if I define

spark

    .readStream

    .format("delta")

    .option("startingVersion", "latest")

    .option("maxFilesPerTrigger ", 100)

Are those the last 100 transactions from the delta transaction history? Or the last 100 parquet files?

Best regards

Michael

jose_gonzalez
Databricks Employee
Databricks Employee

‎07-05-2022 09:59 AM

Hi @Michael Galli​ ,

The maxFilesPerTrigger will measure how many new files to be considered in every micro-batch. The default is 1000. These will be the files associated to your Delta table. So, technically, these will be the parquet files.

0 Kudos

Michael_Galli
Databricks Partner
In response to jose_gonzalez

‎07-05-2022 10:58 PM

Thx @Jose Gonzalez​ , this makes sense..

What I do not fully understand is the role of the Delta table transaction log in this matter.

E.g. maxFilesPerTrigger is set to 100 files for each micro-batch.

If the Delta transactions of the streaming source looks somewhat like this:UnbenanntE.g. there are 70 files per transaction. Will the micro batch 1 contain files from version 0 and 1, micro batch 2 contain files from version 1 and 2, and so on? So the Delta Table version is not really relevant for the streaming micro badge sizing?

Preview file
8 KB
0 Kudos

Vidula
Databricks Partner

‎08-31-2022 01:56 AM

Hey there @Michael Galli​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

0 Kudos