Databricks Community

felix4572 · 09-01-2025

Hello,we employ arbitrary stateful aggregations in our data processing streams on Azure Databricks, and would like to migrate from applyInPandasWithState to transformWithStateInPandas. We employ the Python API throughout our solution, and some of our...

felix4572 · 09-08-2025

Thanks a lot for working on this, @Advika. For now, the workaround to use DBR versions other than 17.1 works for me. Mid-term it would be of course great to use transformWithStateInPandas irrespective of the cluster DBR (as long as the minimum requir...

felix4572 · 09-01-2025

Dear @szymon_dybczak and @-werners- ,thank you a lot for for your responses and references! @-werners- , thank you for the link to the announcement article. The availability section lists that "No-Isolation and Unity Catalog Dedicated Clusters" are s...

felix4572 · 09-01-2025

Dear werners,thank you for your swift response. I use the notebook provided in the example (with a different storage path, of course). The stream config is included.Best!

Databricks Community

User Stats

User Activity

transformWithStateInPandas throws "Spark connect directory is not ready" error

Re: transformWithStateInPandas throws "Spark connect directory is not ready" error

Re: transformWithStateInPandas throws "Spark connect directory is not ready" error

Re: transformWithStateInPandas throws "Spark connect directory is not ready" error