topic maxFilesPerTrigger not working in bronze to silver layer in missing-QuestionPost

maxFilesPerTrigger not working in bronze to silver layer

sanjay — Wed, 14 Jun 2023 09:40:04 GMT

Hi,

I am using Matillion architecture where autoloader picks files from AWS S3 and saves in delta lake. Next layer picks the changes from delta lake and does some processing. I am able to set batch size in autoloader and its working. But in bronze to silver layer, unable to set batch limit, its picking all files in one go. Here is my code from bronze to silver layer..

(spark.readStream.format("delta")

.option("useNotification","true")

.option("includeExistingFiles","true")

.option("allowOverwrites",True)

.option("ignoreMissingFiles",True)

.option("maxFilesPerTrigger", 100)

.load(bronze_path)

.writeStream

.option("checkpointLocation", silver_checkpoint_path)

.trigger(processingTime="1 minute")

.foreachBatch(foreachBatchFunction)

.start()

)

Appreciate any help.

Regards,

Sanjay

Re: maxFilesPerTrigger not working in bronze to silver layer

Anonymous — Fri, 16 Jun 2023 06:08:31 GMT

Hi @Sanjay Jain

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

Re: maxFilesPerTrigger not working in bronze to silver layer

Lakshay — Fri, 16 Jun 2023 11:30:25 GMT

Hi @Sanjay Jain , Could you try using a fresh checkpoint location if not already tried? Also, could you please check the logs what is the size of the micro batch it is currently processing?

Re: maxFilesPerTrigger not working in bronze to silver layer

sanjay — Fri, 16 Jun 2023 11:49:13 GMT

Hi Lakshay,

I tried with new checkpoint location but still its not working. Its taking whole data in one go and not respecting batch size mentioned in code.

Regards,

Sanjay