Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-23-2021 11:13 AM
Does Koalas support Structured Streaming
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-23-2021 11:16 AM
No, Koalas does not support Structured Streaming officially.
As a workaround, you can use Koalas APIs with foreachBatch in Structured Streaming which allows batch APIs:
>>> def func(batch_df, batch_id):
... koalas_df = ks.DataFrame(batch_df)
... koalas_df['a'] = 1
... print(koalas_df)
>>> spark.readStream.format("rate").load().writeStream.foreachBatch(func).start()
timestamp value a
0 2020-02-21 09:49:37.574 4 1
timestamp value a
0 2020-02-21 09:49:38.574 5