https://community.databricks.com/t5/get-started-discussions/spark-handling-white-space-as-null/m-p/65810#M7003 <P>I have a very strange thing happening.&nbsp; I'm importing a csv file and nulls and blanks are being interpreted correctly.&nbsp; What is strange is that a column that regularly has a single space character value is having the single space converted to null.<BR /><BR />I'm using this to import the file data:</P><P>df = spark.read.format("csv").options(mode='FAILFAST', multiLine=True, escape='"').csv(path=source_path, header=True, inferSchema=False).select("*", "_metadata.file_name").withColumns({"date1": lit(currentdt), "GUID": lit(guid)})</P><P>I've tried including option&nbsp;<SPAN>nullValue</SPAN><SPAN>=</SPAN><SPAN>" " but then all blanks are being imported as blank.</SPAN></P><P><SPAN>I'm new to this and sure I'm doing something wrong.</SPAN></P> Mon, 08 Apr 2024 13:41:57 GMT Chrispy 2024-04-08T13:41:57Z https://community.databricks.com/t5/get-started-discussions/spark-handling-white-space-as-null/m-p/65810#M7003 <P>I have a very strange thing happening.&nbsp; I'm importing a csv file and nulls and blanks are being interpreted correctly.&nbsp; What is strange is that a column that regularly has a single space character value is having the single space converted to null.<BR /><BR />I'm using this to import the file data:</P><P>df = spark.read.format("csv").options(mode='FAILFAST', multiLine=True, escape='"').csv(path=source_path, header=True, inferSchema=False).select("*", "_metadata.file_name").withColumns({"date1": lit(currentdt), "GUID": lit(guid)})</P><P>I've tried including option&nbsp;<SPAN>nullValue</SPAN><SPAN>=</SPAN><SPAN>" " but then all blanks are being imported as blank.</SPAN></P><P><SPAN>I'm new to this and sure I'm doing something wrong.</SPAN></P> Mon, 08 Apr 2024 13:41:57 GMT https://community.databricks.com/t5/get-started-discussions/spark-handling-white-space-as-null/m-p/65810#M7003 Chrispy 2024-04-08T13:41:57Z