08-06-2020 03:23 AM
search = search.filter(!F.col("Name").contains("ABC"))
search = search.filter(F.not(F.col("Name").contains("ABC"))
Both methods fail due to syntax error could you please help me filter rows that does not contain a certain string in pyspark.
^ SyntaxError: invalid syntax
08-06-2020 10:34 AM
Here is a complete example
values = [("K1","true","false"),("K2","true","false")] columns = ['Key', 'V1', 'V2'] df = spark.createDataFrame(values, columns) display(df)
FILTER
df2 = df.filter(df.column2 != "delete") display(df2)
08-06-2020 10:35 AM
OOps
Copy and past error, the First code block should be,
values = [("save","save"),("save", "delete")] columns = ['column1','column2'] df = spark.createDataFrame(values, columns) display(df)
