topic Re: Auotoloader-"cloudFiles.backfillInterval" in Get Started Discussions

Auotoloader-"cloudFiles.backfillInterval"

Kiranrathod — Wed, 25 Oct 2023 10:15:28 GMT

1. How to use cloudFiles.backfillInterval option in a notebook?
2. Does It need to be any set of the property?
3. Where is exactly placed readstream portion of the code or writestream portion of the code?
4. Do you have any sample code?
5. Where we find cloudFiles.backfillInterval logs?

Re: Auotoloader-"cloudFiles.backfillInterval"

Kiranrathod — Wed, 01 Nov 2023 11:09:59 GMT

1.Is the following code correct for specifying the .option("cloudFiles.backfillInterval", 300)?
df = spark.readStream.format("cloudFiles") \
.option("cloudFiles.format", "csv") \
.option("cloudFiles.schemaLocation", f"dbfs:/FileStore/xyz/back_fill_option/schema/backfill")\
.load(f"dbfs:/FileStore/xyz/back_fill_option/source")

df.writeStream \
.format("delta") \
.option("cloudFiles.backfillInterval", 300) \
.trigger(processingTime='3 minutes') \
.option("checkpointLocation", f"dbfs:/FileStore/xyz/back_fill_option/checkpoint/backfill") \
.table("back_fill_option")

2.If the autoloader streaming process begins at "2023-11-01T01:00:00" and you set .option("cloudFiles.backfillInterval", 300), does this mean that the backfillInterval will trigger at "2023-11-01T01:05:00"?
3.When you pass the option .trigger(processingTime='3 minutes'), it triggers the process every 3 minutes. If you also set backfillInterval to 2 minutes, does that mean the backfillInterval triggers every 2 minutes?
4.When you set the property processingTime to a value greater than backfillInterval, does that mean the backfillInterval runs before the processingTime interval elapses?
5.How can you verify the functionality of the "cloudFiles.backfillInterval" to ensure it is working correctly with the provided autoloader code?

Re: Auotoloader-"cloudFiles.backfillInterval"

Kiranrathod — Wed, 08 Nov 2023 07:18:31 GMT

Hi @Kaniz , Can you please answer follows question ,
1.Is the following code correct for specifying the .option("cloudFiles.backfillInterval", 300)?
df = spark.readStream.format("cloudFiles") \
.option("cloudFiles.format", "csv") \
.option("cloudFiles.schemaLocation", f"dbfs:/FileStore/xyz/back_fill_option/schema/backfill")\
.load(f"dbfs:/FileStore/xyz/back_fill_option/source")

2.If the autoloader streaming process begins at "2023-11-01T01:00:00" and you set .option("cloudFiles.backfillInterval", 300), does this mean that the backfillInterval will trigger at "2023-11-01T01:05:00"?
3.When you pass the option .trigger(processingTime='3 minutes'), it triggers the process every 3 minutes. If you also set backfillInterval to 2 minutes, does that mean the backfillInterval triggers every 2 minutes?
4.When you set the property processingTime to a value greater than backfillInterval, does that mean the backfillInterval runs before the processingTime interval elapses?
5.How can you verify the functionality of the "cloudFiles.backfillInterval" to ensure it is working correctly with the provided autoloader code?