Does Delta refresh DF cache automatically after a delete?

MoJaMa — Fri, 18 Jun 2021 19:49:41 GMT

Re: Does Delta refresh DF cache automatically after a delete?

MoJaMa — Fri, 18 Jun 2021 19:52:07 GMT

Yes. Delta actually explicitly refreshes the dataframe cache after performing delete.

Use this code to test it out.

Seq((0,"A"), (1,"B"),(2,"C")  ).toDF("id","value").write.format("delta").mode("overwrite").saveAsTable("target_tbl") 
 
val df = spark.sql("select * from target_tbl")
df.persist()
df.show()
 
spark.sql("delete from target_tbl where id = 2")
df.show()
df.unpersist()
 
Output:
+---+-----+
| id|value|
+---+-----+
|  0|    A|
|  2|    C|
|  1|    B|
+---+-----+
+---+-----+
| id|value|
+---+-----+
|  0|    A|
|  1|    B|
+---+-----+

Re: Does Delta refresh DF cache automatically after a delete?

Srikanth_Gupta_ — Fri, 18 Jun 2021 21:11:13 GMT

How about updates and inserts? does it refresh automatically?

topic Re: Does Delta refresh DF cache automatically after a delete? in Data Engineering

Does Delta refresh DF cache automatically after a delete?

Re: Does Delta refresh DF cache automatically after a delete?

Re: Does Delta refresh DF cache automatically after a delete?