Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-22-2024 02:42 AM
Ok, so I recommend to familiar with this documents:
https://docs.databricks.com/en/structured-streaming/delta-lake.html#language-python
https://docs.databricks.com/en/structured-streaming/tutorial.html
Here you can find some sample generic transformation between batch and streaming approach:
# Batch approach:
(spark.read
.table("<table-name1>")
.<some_transformations>
.write
.saveAsTable("<table-name3>")
)
# Streaming approach:
(spark.readStream
.table("<table-name1>")
.<some_transformations>
.writeStream
.trigger(availableNow=True)
.option("checkpointLocation", "<checkpoint-path>")
.saveAsTable("<table-name3>")
)Good luck 🙂