topic Re: Why does df.dropna(how="all") fail when there is a . in a column name? in Data Engineering

Why does df.dropna(how="all") fail when there is a . in a column name?

Abhimanyu — Tue, 20 May 2025 09:39:44 GMT

I'm working in a Databricks notebook and using Spark to query a Delta table. Here's the code I ran:

df = spark.sql("select * from catalog.schema.table") df = df.dropna(how="all") display(df)

This works fine unless the DataFrame has a column name that contains a dot (.), like Disc.. When such a column exists, the dropna(how="all") line fails with a syntax error saying:

Syntax error in attribute name: Disc.

I understand that . has a special meaning in Spark SQL (used for nested fields), but why does this cause an issue with a general dropna() operation that doesn’t reference column names explicitly?

Is this a known limitation or bug?
What's the best way to handle such cases where column names have dots?
Should I always rename such columns before transformation?
Thanks in advance!

Re: Why does df.dropna(how="all") fail when there is a . in a column name?

MujtabaNoori — Tue, 20 May 2025 12:55:54 GMT

Hi @Abhimanyu ,
Yeah Actually in spark, '.'(dot) in columns is used for StructType by which the nested object can be accessed.
But definitely You can rename thE columns dynamically whichever has '.' in it.
Attached few screenshots for your reference that demonstrates the dropna(how='all') functionality for '.' columns without manual work

CODE FOR YOUR REFERENCE

cols = []

for col in df.columns:

if '.' in col:

renamedCol = col.replace('.','_')

cols.append(f"`{col}` as {renamedCol}")

continue

cols.append(col)

df = df.selectExpr(*cols)

Re: Why does df.dropna(how="all") fail when there is a . in a column name?

Abhimanyu — Mon, 09 Jun 2025 09:31:37 GMT

but you actually renamed the column and put an underscore '_' instead of a dot '.'