topic Filtering rows that does not contain a string in Data Engineering

Filtering rows that does not contain a string

KutayKoralturk — Thu, 06 Aug 2020 10:23:47 GMT

search = search.filter(!F.col("Name").contains("ABC"))

search = search.filter(F.not(F.col("Name").contains("ABC"))

Both methods fail due to syntax error could you please help me filter rows that does not contain a certain string in pyspark.

^ SyntaxError: invalid syntax

User16857282152 — Thu, 06 Aug 2020 17:34:20 GMT

Here is a complete example

values = [("K1","true","false"),("K2","true","false")] 
columns = ['Key', 'V1', 'V2']
df = spark.createDataFrame(values, columns)
display(df)

FILTER

df2 = df.filter(df.column2 != "delete")
display(df2)

User16857282152 — Thu, 06 Aug 2020 17:35:23 GMT

OOps

Copy and past error, the First code block should be,

values = [("save","save"),("save", "delete")] 
columns = ['column1','column2']
df = spark.createDataFrame(values, columns)
display(df)