topic training_set.load_df().toPandas() fails with the new pandas version (2.0.0) in Data Engineering

training_set.load_df().toPandas() fails with the new pandas version (2.0.0)

ALIDI — Tue, 04 Apr 2023 08:40:12 GMT

pandas 2.0.0 was released on 4.3.2023 and was pushed to my cluster on the same day.

The day after I tried using training_set.load_df().toPandas() and it failed. Reverting to pandas 1.5.3. fixed the problem.

Re: training_set.load_df().toPandas() fails with the new pandas version (2.0.0)

Anonymous — Mon, 10 Apr 2023 13:09:03 GMT

@Al IDI :

It is possible that there are some compatibility issues between the current version of Databricks and the new version of Pandas. You may need to wait for an update or patch from Databricks to address this issue. In the meantime, you can continue using Pandas version 1.5.3, or try downgrading to a different version that is compatible with Databricks. You can also try using a different method to load the data, such as spark.createDataFrame(), or spark.read.format() to read the data directly into a Spark DataFrame and then convert it to a Pandas DataFrame.

Re: training_set.load_df().toPandas() fails with the new pandas version (2.0.0)

Anonymous — Wed, 12 Apr 2023 07:54:20 GMT

Hi @Al IDI

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!

Re: training_set.load_df().toPandas() fails with the new pandas version (2.0.0)

ALIDI — Thu, 13 Apr 2023 05:55:07 GMT

Thanks for the update. Using a different method is problematic since I take advantage of the feature_store.score_batch() method in my production pipeline.

How to you recommend following up on this specific future update from databricks?