topic Re: PythonException: TypeError: float() argument must be a string or a number, not 'NoneType' in Data Engineering

PythonException: TypeError: float() argument must be a string or a number, not 'NoneType'

kll — Fri, 07 Jul 2023 18:13:54 GMT

I get an PythonException: float() argument must be a string or a number, not 'NoneType' when attempting to save a DataFrame as a Delta Table.

Here's the line of code that I am running:

```

df.write.format("delta").saveAsTable("schema1.df_table", mode="overwrite")

root

|-- ts: timestamp (nullable = true)

|-- _source: string (nullable = true)

|-- lat: decimal(9,6) (nullable = true)

|-- lng: decimal(9,6) (nullable = true)

|-- id: string (nullable = false)

|-- mm-yyyy: date (nullable = true)

|-- hid: string (nullable = true)

```

PythonException: An exception was thrown from the Python worker. Please see the stack trace below. 'TypeError: float() argument must be a string or a number, not 'NoneType'', from , line 3. Full traceback below: Traceback (most recent call last): File "", line 3, in TypeError: float() argument must be a string or a number, not 'NoneType'

How do we handle `None` in the spark DataFrame? I'd like to identify which rows / columns contain `None` and drop them?

Re: PythonException: TypeError: float() argument must be a string or a number, not 'NoneType'

shan_chandra — Fri, 07 Jul 2023 21:49:01 GMT

@kll - can you please try using try/catch option to convert None to an unique value for the decimal columns.

Re: PythonException: TypeError: float() argument must be a string or a number, not 'NoneType'

shan_chandra — Fri, 07 Jul 2023 21:49:53 GMT

something like the below

try: if lng is not None: new_lng = lng else: new_lng = 0

Re: PythonException: TypeError: float() argument must be a string or a number, not 'NoneType'

Anonymous — Wed, 12 Jul 2023 09:07:39 GMT

Hi @kll

Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.

Cheers!

Re: PythonException: TypeError: float() argument must be a string or a number, not 'NoneType'

Lakshay — Thu, 13 Jul 2023 16:30:37 GMT

Even though the code throws the issue while write, the issue can be in the code before as spark is lazily evaluated. The error "TypeError: float() argument must be a string or a number, not 'NoneType'" generally comes when we pass a variable to float() function and the value of the variable at that time is None. I would recommend you to check for such issues.