What’s the easiest way to clean and transform data using PySpark in Databricks?

[Suheb]

You have some raw data (like messy Excel files, CSVs, or logs) and you want to prepare it for analysis — by removing errors, fixing missing values, changing formats, or combining columns — using PySpark (Python for Apache Spark) inside Databricks.

[szymon_dybczak]

Hi @Suheb ,

You can do almost anything with data using Pyspark. Depending what you want to achieve you can :

- do you want to remove duplicates?

df = df.dropDuplicates()

- do you want to change types? 

df = df.withColumn("sales", col("sales").cast("double"))
df = df.withColumn("date", col("date").cast("date"))

- do you want handle missing data?

df = df.fillna({'city': 'Unknown', 'sales': 0})
# or drop rows
df = df.na.drop(subset=['id', 'sales'])

 - or maybe you want apply some transformation?

from pyspark.sql.functions import trim, upper

df = df.withColumn("country", upper(trim(col("country"))))

And many more. You can do virtually anything with data using pyspark 🙂

[ShaneCorn]

The easiest way to clean and transform data using PySpark in Databricks is by leveraging the DataFrame API. Start by loading data into a Spark DataFrame with spark.read. Use built-in functions like dropna, fillna, and withColumn to handle missing values and create new columns. Apply filter or select for subsetting data, and use groupBy with aggregation for summaries. Databricks’ interactive notebooks make it easy to visualize results instantly. Finally, write the cleaned data back to storage using df.write in formats like Parquet or Delta Lake.