11-10-2021 11:26 AM
To make the comparisons case insensitive we can compare after converting columns to upper or lower case.
Eg:
import org.apache.spark.sql.functions._
df.filter(lower($"colname") == lower($"colname"))
