Databricks Community

SohelKhan · 02-22-2016

Pyspark 1.6: DataFrame: Converting one column from string to float/double I have two columns in a dataframe both of which are loaded as string. DF = rawdata.select('house name', 'price') I want to convert DF.price to float. DF = rawdata.select('hous...

SohelKhan · 02-21-2016

In SQL select, in some implementation, we can provide select -col_A to select all columns except the col_A. I tried it in the Spark 1.6.0 as follows: For a dataframe df with three columns col_A, col_B, col_C df.select('col_B, 'col_C') # it works df....

SohelKhan · 02-28-2016

Thanks for the suggestion. Sorry though, it did not work. from pyspark.sql.functions import udf sqlContext.udf.register("float",lambda x:float(x)) from pyspark.sql.functions import expr DF = rawdata.select('name', expr(float('numberfloat'))) ----...

SohelKhan · 02-27-2016

I fixed it as follows: from pyspark.sql.functions import udf from pyspark.sql.types import StringType def string_to_float(x): return float(x) udfstring_to_float = udf(string_to_float, StringType()) rawdata.withColumn("name",udfstring_t...

Databricks Community

User Stats

User Activity

Pyspark DataFrame: Converting one column from string to float/double

PySpark DataFrame: Select all but one or a set of columns

Re: Pyspark DataFrame: Converting one column from string to float/double

Re: Pyspark DataFrame: Converting one column from string to float/double