Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-18-2023 04:43 PM - edited 07-18-2023 04:44 PM
this wont work if you are creating a table for the first time from the stream, for example the code below when running for the first time. I need a way to capture the file name going in the stream
# Configure Auto Loader
streaming_query = (spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "csv")
.option("cloudFiles.schemaLocation", raw_checkpoint_path)
.option("sep", "|")
.option("inferSchema", "true")
.option("lineSep", "\r\n") # Specify the Windows-style EOL character (CRLF)
.option("pathGlobfilter", file_pattern)
.load(f"{file_path}")
.select("*", input_file_name().alias("source_file"), current_timestamp().alias("processing_time"))
.writeStream
.option("checkpointLocation", raw_checkpoint_path)
.trigger(availableNow=True)
.toTable(raw_table_name))
