cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Warehousing & Analytics
Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Community. Share insights, tips, and best practices for leveraging data for informed decision-making.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Does Koalas support Structured Streaming

User16826994223
Honored Contributor III

Does Koalas support Structured Streaming

1 ACCEPTED SOLUTION

Accepted Solutions

User16826994223
Honored Contributor III

No, Koalas does not support Structured Streaming officially.

As a workaround, you can use Koalas APIs with foreachBatch in Structured Streaming which allows batch APIs:

>>> def func(batch_df, batch_id):
 
...     koalas_df = ks.DataFrame(batch_df)
 
...     koalas_df['a'] = 1
 
...     print(koalas_df)
 
โ€‹
 
>>> spark.readStream.format("rate").load().writeStream.foreachBatch(func).start()
 
                timestamp  value  a
 
0 2020-02-21 09:49:37.574      4  1
 
                timestamp  value  a
 
0 2020-02-21 09:49:38.574      5  

View solution in original post

1 REPLY 1

User16826994223
Honored Contributor III

No, Koalas does not support Structured Streaming officially.

As a workaround, you can use Koalas APIs with foreachBatch in Structured Streaming which allows batch APIs:

>>> def func(batch_df, batch_id):
 
...     koalas_df = ks.DataFrame(batch_df)
 
...     koalas_df['a'] = 1
 
...     print(koalas_df)
 
โ€‹
 
>>> spark.readStream.format("rate").load().writeStream.foreachBatch(func).start()
 
                timestamp  value  a
 
0 2020-02-21 09:49:37.574      4  1
 
                timestamp  value  a
 
0 2020-02-21 09:49:38.574      5  

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group