topic Re: How do I cast using a DataFrame? in Data Engineering

How do I cast using a DataFrame?

cfregly — Tue, 28 Apr 2015 20:03:12 GMT

Re: How do I cast using a DataFrame?

cfregly — Tue, 28 Apr 2015 20:06:49 GMT

You can use HiveQL's cast() type conversion function to cast an element of a nested map in Python as follows:

from pyspark.sql import Row 
df = sqlContext.createDataFrame([Row(a={'b': 1})])
str = df.selectExpr("cast(a['b'] AS STRING)")

or in Scala as follows:

val df = Seq((Map("a" -> 1))).toDF("a") 
df.selectExpr("cast(a['a'] AS STRING)")

Re: How do I cast using a DataFrame?

Grr — Wed, 01 Feb 2017 15:57:51 GMT

If your df is registered as a table you can also do this with a SQL call:

df.createOrReplaceTempView("table")
str = spark.sql('''
    SELECT CAST(a['b'] AS STRING)
    FROM table
''')

Its more code in the simple case but I have found in the past that when this is combined into a much more complex query the SQL format can be more friendly from a readability standpoint.

Re: How do I cast using a DataFrame?

DarrellUlm — Wed, 15 Mar 2017 19:26:02 GMT

Could also use withColumn() to do it without Spark-SQL, although the performance will likely be different. The question being, would creating a new column take more time than using Spark-SQL.

Something like:

val dfNew = df.withColumn("newColName", df.originalColName.cast(IntegerType))
    .drop("originalColName").withColumnRenamed("newColName", "originalColName")

Create the new column, casting from the original column, drop the original, then rename the new column back to the original name. A bit roundabout, but looks like it could work.

Re: How do I cast using a DataFrame?

ShubhamGupta187 — Fri, 20 Apr 2018 04:33:31 GMT

Is it safe to cast a column that contains null values?

Re: How do I cast using a DataFrame?

srisre111 — Thu, 19 Mar 2020 14:24:38 GMT

I am trying to store a dataframe as table in databricks and encountering the following error, can someone help?

"typeerror: field date: can not merge type <class 'pyspark.sql.types.stringtype'> and <class 'pyspark.sql.types.doubletype'>"