Why does df.dropna(how="all") fail when there is a . in a column name?

Abhimanyu
Databricks Partner

‎05-20-2025 02:39 AM

I'm working in a Databricks notebook and using Spark to query a Delta table. Here's the code I ran:

 
df = spark.sql("select * from catalog.schema.table") df = df.dropna(how="all") display(df)

This works fine unless the DataFrame has a column name that contains a dot (.), like Disc.. When such a column exists, the dropna(how="all") line fails with a syntax error saying:

 
Syntax error in attribute name: Disc.

I understand that . has a special meaning in Spark SQL (used for nested fields), but why does this cause an issue with a general dropna() operation that doesn’t reference column names explicitly?

  • Is this a known limitation or bug?

  • What's the best way to handle such cases where column names have dots?

  • Should I always rename such columns before transformation?

    Thanks in advance!

0 Kudos

MujtabaNoori
New Contributor III

‎05-20-2025 05:55 AM

Hi @Abhimanyu ,
Yeah Actually in spark, '.'(dot) in columns is used for StructType by which the nested object can be accessed.
But definitely You can rename thE columns dynamically whichever has '.' in it.
Attached few screenshots for your reference that demonstrates the dropna(how='all') functionality for '.' columns without manual work

MujtabaNoori_0-1747745580276.png

 

MujtabaNoori_1-1747745586077.png

 

CODE FOR YOUR REFERENCE

cols = []
for col in df.columns:
    if '.' in col:
        renamedCol = col.replace('.','_')
        cols.append(f"`{col}` as {renamedCol}")
        continue
    cols.append(col)
df = df.selectExpr(*cols)
0 Kudos

Abhimanyu
Databricks Partner
In response to MujtabaNoori

‎06-09-2025 02:31 AM

but you actually renamed the column and put an underscore '_' instead of a dot '.'

0 Kudos