Why does df.dropna(how="all") fail when there is a . in a column name?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
05-20-2025 02:39 AM
I'm working in a Databricks notebook and using Spark to query a Delta table. Here's the code I ran:
df = spark.sql("select * from catalog.schema.table") df = df.dropna(how="all") display(df)
This works fine unless the DataFrame has a column name that contains a dot (.), like Disc.. When such a column exists, the dropna(how="all") line fails with a syntax error saying:
Syntax error in attribute name: Disc.
I understand that . has a special meaning in Spark SQL (used for nested fields), but why does this cause an issue with a general dropna() operation that doesn’t reference column names explicitly?
Is this a known limitation or bug?
What's the best way to handle such cases where column names have dots?
Should I always rename such columns before transformation?
Thanks in advance!
