Hi
I am using auto loader to fetch some records stored in two files. Please see below my code. It fetches records from two files correctly and then it starts fetching NULL records. I attach option("cleanSource", ) to readStream. But it is not working and does not create directory "/FileStore/archive" as shown below. Please help me.
schema = "user_id long, device_id long, mac_address string, registration_timestamp double"
(spark.readStream
.format("cloudFiles")
.schema(schema)
.option("cloudFiles.format","csv")
.option("maxFilesPerTrigger", 1)
.option("header", True)
.option("cleanSource", "archive")
.option("sourceArchiveDir", "/FileStore/archive")
.option("mergeSchema",True)
.load("/FileStore/auto_loader")
.writeStream
.format("delta")
.option("checkpointLocation", "/FileStore/checkpoint")
.outputMode("append")
.toTable("tempdb.bzTable")
)
