Auotoloader-"cloudFiles.backfillInterval"

Kiranrathod
New Contributor III

‎10-25-2023 03:15 AM

1. How to use cloudFiles.backfillInterval option in a notebook?
2. Does It need to be any set of the property?
3. Where is exactly placed readstream portion of the code or writestream portion of the code?
4. Do you have any sample code?
5. Where we find cloudFiles.backfillInterval logs?

 

 

 

Preview file
103 KB
0 Kudos

Kiranrathod
New Contributor III
In response to Retired_mod

‎11-07-2023 11:18 PM

Hi @Kaniz , Can you please answer follows question ,
1.Is the following code correct for specifying the .option("cloudFiles.backfillInterval", 300)?
df = spark.readStream.format("cloudFiles") \
.option("cloudFiles.format", "csv") \
.option("cloudFiles.schemaLocation", f"dbfs:/FileStore/xyz/back_fill_option/schema/backfill")\
.load(f"dbfs:/FileStore/xyz/back_fill_option/source")

df.writeStream \
.format("delta") \
.option("cloudFiles.backfillInterval", 300) \
.trigger(processingTime='3 minutes') \
.option("checkpointLocation", f"dbfs:/FileStore/xyz/back_fill_option/checkpoint/backfill") \
.table("back_fill_option")

2.If the autoloader streaming process begins at "2023-11-01T01:00:00" and you set .option("cloudFiles.backfillInterval", 300), does this mean that the backfillInterval will trigger at "2023-11-01T01:05:00"?
3.When you pass the option .trigger(processingTime='3 minutes'), it triggers the process every 3 minutes. If you also set backfillInterval to 2 minutes, does that mean the backfillInterval triggers every 2 minutes?
4.When you set the property processingTime to a value greater than backfillInterval, does that mean the backfillInterval runs before the processingTime interval elapses?
5.How can you verify the functionality of the "cloudFiles.backfillInterval" to ensure it is working correctly with the provided autoloader code?

0 Kudos

Kiranrathod
New Contributor III

‎11-01-2023 04:09 AM

1.Is the following code correct for specifying the  .option("cloudFiles.backfillInterval", 300)?
df = spark.readStream.format("cloudFiles") \
.option("cloudFiles.format", "csv") \
.option("cloudFiles.schemaLocation", f"dbfs:/FileStore/xyz/back_fill_option/schema/backfill")\
.load(f"dbfs:/FileStore/xyz/back_fill_option/source")

df.writeStream \
.format("delta") \
.option("cloudFiles.backfillInterval", 300) \
.trigger(processingTime='3 minutes') \
.option("checkpointLocation", f"dbfs:/FileStore/xyz/back_fill_option/checkpoint/backfill") \
.table("back_fill_option")

2.If the autoloader streaming process begins at "2023-11-01T01:00:00" and you set .option("cloudFiles.backfillInterval", 300), does this mean that the backfillInterval will trigger at "2023-11-01T01:05:00"?
3.When you pass the option .trigger(processingTime='3 minutes'), it triggers the process every 3 minutes. If you also set backfillInterval to 2 minutes, does that mean the backfillInterval triggers every 2 minutes?
4.When you set the property processingTime to a value greater than backfillInterval, does that mean the backfillInterval runs before the processingTime interval elapses?
5.How can you verify the functionality of the "cloudFiles.backfillInterval" to ensure it is working correctly with the provided autoloader code?

0 Kudos