cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

databricks-connect library for python and pandas 3

Albertino
New Contributor II

Hello,

databricks-connect is pinning pandas โค๏ธ during the installation. Since we're moving towards pandas 3 can you please add the support for the newest version as well?

3 REPLIES 3

Brahmareddy
Esteemed Contributor II

Hi @Albertino , how are you doing today?

Thanks for calling this out. as per my understanding, at the moment, the pandas pin is there intentionally. Databricks Connect release notes say supported pandas versions are currently limited to 1.0.5<=pandas<3 to avoid breakage in pyspark.pandas workloads after the pandas 3.0.0 release.

Fully agree that pandas 3 support will matter. It likely needs proper compatibility work across Databricks Connect and pandas API on Spark before the restriction can be lifted safely. Hopefully that lands soon.

SergejK
Visitor

Hello, is there a timeframe when the pandas 3.0 support will be implemented?

Brahmareddy
Esteemed Contributor II

Hi @SergejK,

Good question! As of now, I haven't seen an official ETA from Databricks for pandas 3.0 support in Databricks Connect.

The current restriction (pandas<3) was introduced to prevent compatibility issues with the pandas API on Spark (pyspark.pandas).

For now, I would recommend staying on pandas 2.x for Databricks Connect workloads. Hopefully, the Databricks team can share an estimated timeline, as more teams are moving toward pandas 3.0.

You can track updates here: https://docs.databricks.com/aws/en/release-notes/dbconnect

Hope this helps!