Structured Streaming migration from readStream to Auto Loader (AWS)

Cosimo_F_
Contributor

Hello!

I have a (non-stateful) daily structured streaming ingestion job from json S3 file storage into a Delta table using Spark readStream. I would like to switch to Auto Loader to reduce file discovery time. My questions are:

  • Do I need to create a new streaming checkpoint? (I am guessing the answer to this question is yes)
  • Can I keep appending to the same Delta table (and table location), or do I need to create a new table as well?

Thank you!

Cosimo.