cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

PythonException: TypeError: float() argument must be a string or a number, not 'NoneType'

kll
New Contributor III

I get an PythonException: float() argument must be a string or a number, not 'NoneType' when attempting to save a DataFrame as a Delta Table. 

Here's the line of code that I am running:

```

df.write.format("delta").saveAsTable("schema1.df_table", mode="overwrite")

root

|-- ts: timestamp (nullable = true)

|-- _source: string (nullable = true)

|-- lat: decimal(9,6) (nullable = true)

|-- lng: decimal(9,6) (nullable = true)

|-- id: string (nullable = false)

|-- mm-yyyy: date (nullable = true)

|-- hid: string (nullable = true)

```

 

PythonException: An exception was thrown from the Python worker. Please see the stack trace below. 'TypeError: float() argument must be a string or a number, not 'NoneType'', from , line 3. Full traceback below: Traceback (most recent call last): File "", line 3, in TypeError: float() argument must be a string or a number, not 'NoneType'

 

How do we handle `None` in the spark DataFrame? I'd like to identify which rows / columns contain `None` and drop them? 

4 REPLIES 4

shan_chandra
Esteemed Contributor
Esteemed Contributor

@kll - can you please try using try/catch option to convert None to an unique value for the decimal columns. 

something like the below 

try:
if lng is not None:
new_lng = lng
else:
new_lng = 0

 

Anonymous
Not applicable

Hi @kll 

Hope you are well. Just wanted to see if you were able to find an answer to your question and would you like to mark an answer as best? It would be really helpful for the other members too.

Cheers!

Lakshay
Esteemed Contributor
Esteemed Contributor

Even though the code throws the issue while write, the issue can be in the code before as spark is lazily evaluated. The error "TypeError: float() argument must be a string or a number, not 'NoneType'" generally comes when we pass a variable to float() function and the value of the variable at that time is None. I would recommend you to check for such issues.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!