chanukya-pekala
Contributor III

Thanks for discussion. I have a tiny suggestion. Based on my experience working with streaming loads, I often find the checkpoint location hard enough to actually check the offset information or delete that directory for fresh load of data. Hence I have used TBLPROPERTIES ('checkpointLocation' = '/path/to/checkpoint') by altering the table. Any user will then be able to know where is checkpoint of this particular table. Databricks/Spark by principle do not want to tightly couple the checkpoint with the table for scalability purposes but having it in some reference to table makes more sense, Without having this critical piece of info, table user needs to reach out to developer only to get the location.

Chanukya

View solution in original post