Re: maxFilesPerTrigger not working in bronze to si...

sanjay · ‎06-14-2023

Hi,

I am using Matillion architecture where autoloader picks files from AWS S3 and saves in delta lake. Next layer picks the changes from delta lake and does some processing. I am able to set batch size in autoloader and its working. But in bronze to silver layer, unable to set batch limit, its picking all files in one go. Here is my code from bronze to silver layer..

(spark.readStream.format("delta")

.option("useNotification","true")

.option("includeExistingFiles","true")

.option("allowOverwrites",True)

.option("ignoreMissingFiles",True)

.option("maxFilesPerTrigger", 100)

.load(bronze_path)

.writeStream

.option("checkpointLocation", silver_checkpoint_path)

.trigger(processingTime="1 minute")

.foreachBatch(foreachBatchFunction)

.start()

)

Appreciate any help.

Regards,

Sanjay

Anonymous · ‎06-15-2023

Hi @Sanjay Jain

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

Lakshay · ‎06-16-2023

Hi @Sanjay Jain , Could you try using a fresh checkpoint location if not already tried? Also, could you please check the logs what is the size of the micro batch it is currently processing?

sanjay · ‎06-16-2023

Hi Lakshay,

I tried with new checkpoint location but still its not working. Its taking whole data in one go and not respecting batch size mentioned in code.

Regards,

Sanjay

maxFilesPerTrigger not working in bronze to silver layer