Does Koalas support Structured Streaming

User16826994223 — Wed, 23 Jun 2021 18:13:50 GMT

Re: Does Koalas support Structured Streaming

User16826994223 — Wed, 23 Jun 2021 18:16:18 GMT

No, Koalas does not support Structured Streaming officially.

As a workaround, you can use Koalas APIs with foreachBatch in Structured Streaming which allows batch APIs:

>>> def func(batch_df, batch_id):
 
...     koalas_df = ks.DataFrame(batch_df)
 
...     koalas_df['a'] = 1
 
...     print(koalas_df)
 

 
>>> spark.readStream.format("rate").load().writeStream.foreachBatch(func).start()
 
                timestamp  value  a
 
0 2020-02-21 09:49:37.574      4  1
 
                timestamp  value  a
 
0 2020-02-21 09:49:38.574      5

topic Does Koalas support Structured Streaming in Warehousing & Analytics

Does Koalas support Structured Streaming

Re: Does Koalas support Structured Streaming