Why does df.dropna(how="all") fail when there is a . in a column name?

[Abhimanyu]

I'm working in a Databricks notebook and using Spark to query a Delta table. Here's the code I ran:

 

df = spark.sql("select * from catalog.schema.table") df = df.dropna(how="all") display(df)

This works fine unless the DataFrame has a column name that contains a dot (.), like Disc.. When such a column exists, the dropna(how="all") line fails with a syntax error saying:

 

Syntax error in attribute name: Disc.

I understand that . has a special meaning in Spark SQL (used for nested fields), but why does this cause an issue with a general dropna() operation that doesn’t reference column names explicitly?

• Is this a known limitation or bug?

• What's the best way to handle such cases where column names have dots?

• Should I always rename such columns before transformation?

  Thanks in advance!

[MujtabaNoori]

Hi @Abhimanyu ,
Yeah Actually in spark, '.'(dot) in columns is used for StructType by which the nested object can be accessed.
But definitely You can rename thE columns dynamically whichever has '.' in it.
Attached few screenshots for your reference that demonstrates the dropna(how='all') functionality for '.' columns without manual work

CODE FOR YOUR REFERENCE

cols = []

for col in df.columns:

    if '.' in col:

        renamedCol = col.replace('.','_')

        cols.append(f"`{col}` as {renamedCol}")

        continue

    cols.append(col)

df = df.selectExpr(*cols)

[Abhimanyu]

but you actually renamed the column and put an underscore '_' instead of a dot '.'