When should I use ".start()" with writeStream?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-20-2022 12:44 AM
Hi,
I am practicing with Databricks. In sample notebooks,I have seen different use of writeStream with or without ".start()" method. Samples are below:
Without .start()
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", source_format)
.option("cloudFiles.schemaLocation", checkpoint_directory)
.load(data_source)
.writeStream
.option("checkpointLocation", checkpoint_directory)
.option("mergeSchema", "true")
.table(table_name)
With .start()
(myDF
.writeStream
.format("delta")
.option("checkpointLocation", checkpointPath)
.outputMode("append")
.start(path)
)
With .start()
query = (streaming_df.writeStream
.foreachBatch(streaming_merge.upsert_to_delta)
.outputMode("update")
.option("checkpointLocation", f"{DA.paths.checkpoints}/recordings")
.trigger(availableNow=True)
.start())
query.awaitTermination()
1) I didn't understand where should / shouldn't use ".start()" method. I appreciate it if you could guide me on this.
2) If I don't pass "path" to the "start()", where the data files will be written?
Thanks for your help.
- Labels:
-
Path
-
Start
-
Streaming
-
WrtieStream
![](/skins/images/1C7D039E274DA4E433FB1B1A3EAE173A/responsive_peak/images/icon_anonymous_profile.png)
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-27-2022 05:46 AM
Hi @Mohammad Saber
Great to meet you, and thanks for your question!
Let's see if your peers in the community have an answer to your question first. Or else bricksters will get back to you soon.
Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-27-2022 12:39 PM
Thanks for your message. I am still looking for the answer.
![](/skins/images/B38AF44D4BD6CE643D2A527BE673CCF6/responsive_peak/images/icon_anonymous_message.png)
![](/skins/images/B38AF44D4BD6CE643D2A527BE673CCF6/responsive_peak/images/icon_anonymous_message.png)