cancel
Showing results for 
Search instead for 
Did you mean: 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results for 
Search instead for 
Did you mean: 

Does Koalas support Structured Streaming

User16826994223
Honored Contributor III

Does Koalas support Structured Streaming

1 ACCEPTED SOLUTION

Accepted Solutions

User16826994223
Honored Contributor III

No, Koalas does not support Structured Streaming officially.

As a workaround, you can use Koalas APIs with foreachBatch in Structured Streaming which allows batch APIs:

>>> def func(batch_df, batch_id):
 
...     koalas_df = ks.DataFrame(batch_df)
 
...     koalas_df['a'] = 1
 
...     print(koalas_df)
 
​
 
>>> spark.readStream.format("rate").load().writeStream.foreachBatch(func).start()
 
                timestamp  value  a
 
0 2020-02-21 09:49:37.574      4  1
 
                timestamp  value  a
 
0 2020-02-21 09:49:38.574      5  

View solution in original post

1 REPLY 1

User16826994223
Honored Contributor III

No, Koalas does not support Structured Streaming officially.

As a workaround, you can use Koalas APIs with foreachBatch in Structured Streaming which allows batch APIs:

>>> def func(batch_df, batch_id):
 
...     koalas_df = ks.DataFrame(batch_df)
 
...     koalas_df['a'] = 1
 
...     print(koalas_df)
 
​
 
>>> spark.readStream.format("rate").load().writeStream.foreachBatch(func).start()
 
                timestamp  value  a
 
0 2020-02-21 09:49:37.574      4  1
 
                timestamp  value  a
 
0 2020-02-21 09:49:38.574      5  

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group