topic Error of "name 'IntegerType' is not defined" in attempting to convert a DF column to IntegerType in Data Engineering

Error of "name 'IntegerType' is not defined" in attempting to convert a DF column to IntegerType

BingQian — Sun, 24 May 2020 05:00:55 GMT

initialDF .withColumn("OriginalCol", initialDF.OriginalCol.cast(IntegerType))

initialDF .withColumn("OriginalCol", initialDF.OriginalCol.cast(IntegerType()))

However, always failed with this error :

NameError: name 'IntegerType' is not defined

Any help is appreciated.

KristoRaun — Sun, 24 May 2020 21:03:43 GMT

I'm assuming you are using Python.

In order to use the IntegerType, you first have to import it with the following statement:

from pyspark.sql.types import IntegerType

If you plan to have various conversions, it will make sense to import all types. This can be done as follows:

from pyspark.sql.types import *

For further reference, you can check the Spark documentation:

BingQian — Mon, 25 May 2020 01:50:09 GMT

Thank you @Kristo Raun !