brockb
Databricks Employee
Databricks Employee

Hi @abueno ,

To replicate a SQL `not like '%var4%'` clause in the Dataframe API, you could use `rlike` with negation using `~` such as:

df.filter(~col('col4').rlike('var4')).display()

Here's a basic reproducible example:

df = (spark.range(10).withColumn("col4", f.lit("var3"))).union(
      spark.range(10).withColumn("col4", f.lit("var4")))

df.filter(~col('col4').rlike('var4')).groupBy('col4').count().display()
col4	count
var3	10

Hope this helps.

View solution in original post