Databricks Community

enri_casca · 03-01-2022

Hi, I am very new in databricks and I am trying to run quick experiments to understand the best practice for me, my colleagues and the company.I pull the data from snowflakedf = spark.read \ .format("snowflake") \ .options(**options) \ .option('qu...

enri_casca · 03-02-2022

I can add that if I convert the data type in sparkif I use toPandas() --> then it worksif I use to_pandas_on_spark() -->same error

enri_casca · 03-02-2022

This is the weird thing. The column is already being transformed in float and you can see that when you call dtypes, so if I try to do one of these methods to check commas or anything else it says"Cannot call StringMethods on type FloatType"but the s...

enri_casca · 03-01-2022

Ok, understood the trasformation to pandas, thank you :). But since I had everything in an object format I always converted all the columns to the correct format using astype(format)so when I run df.dtypes I see the correct format but still when I tr...

enri_casca · 03-01-2022

the schema of the spark dataframe is perfectly fine with all the features different (date, string, decimal)

enri_casca · 03-01-2022

it is a pyspark.sqldataframe.dataframeto convert to pandas I have tried withdf.to_pandas_on_spark()df.toPandas()and import pyspark.pandas as psps.DataFrame(df)all of them same result with everything becoming an object.but at the same time why also af...

Databricks Community

User Stats

User Activity

Couldn't convert string to float when fit model

Re: Couldn't convert string to float when fit model

Re: Couldn't convert string to float when fit model

Re: Couldn't convert string to float when fit model

Re: Couldn't convert string to float when fit model

Re: Couldn't convert string to float when fit model