Databricks Community

ckarrasexo · 05-29-2024

I noticed that on some Databricks 14.3 clusters, I get DataFrames with type pyspark.sql.connect.dataframe.DataFrame, while on other clusters also with Databricks 14.3, the exact same code gets DataFrames of type pyspark.sql.DataFramepyspark.sql.conne...

ckarrasexo · 05-31-2024

Hi @Retired_mod, That's incorrect. I use exactly the same code and either get a pyspark.sql.dataframe.DataFrame, or pyspark.sql.connect.dataframe.DataFrame depending on the cluster. It doesn't matter if I create the dataframe using spark.read.table, ...

Databricks Community

User Stats

User Activity

pyspark.sql.connect.dataframe.DataFrame vs pyspark.sql.DataFrame

Re: pyspark.sql.connect.dataframe.DataFrame vs pyspark.sql.DataFrame