topic Re: Streaming with Delta table source- definition of "File"? in Data Engineering

Streaming with Delta table source- definition of "File"?

Michael_Galli — Mon, 04 Jul 2022 09:11:21 GMT

Hi all,

I have a Delta Table as a Spark Streaming source.

This table contains signals on row level -> each signal is one append to the source table that creates a new version in the delta transaction history.

I am not really sure now how Spark streaming works if I define

spark

.readStream

.format("delta")

.option("startingVersion", "latest")

.option("maxFilesPerTrigger ", 100)

Are those the last 100 transactions from the delta transaction history? Or the last 100 parquet files?

Best regards

Michael

Re: Streaming with Delta table source- definition of "File"?

jose_gonzalez — Tue, 05 Jul 2022 16:59:14 GMT

Hi @Michael Galli ,

The maxFilesPerTrigger will measure how many new files to be considered in every micro-batch. The default is 1000. These will be the files associated to your Delta table. So, technically, these will be the parquet files.

Re: Streaming with Delta table source- definition of "File"?

Michael_Galli — Wed, 06 Jul 2022 05:58:17 GMT

Thx @Jose Gonzalez , this makes sense..

What I do not fully understand is the role of the Delta table transaction log in this matter.

E.g. maxFilesPerTrigger is set to 100 files for each micro-batch.

If the Delta transactions of the streaming source looks somewhat like this:E.g. there are 70 files per transaction. Will the micro batch 1 contain files from version 0 and 1, micro batch 2 contain files from version 1 and 2, and so on? So the Delta Table version is not really relevant for the streaming micro badge sizing?

Re: Streaming with Delta table source- definition of "File"?

Vidula — Wed, 31 Aug 2022 08:56:50 GMT

Hey there @Michael Galli

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!