https://community.databricks.com/t5/data-engineering/structured-streaming-migration-from-readstream-to-auto-loader/m-p/13129#M7848 <P>Hello!</P><P></P><P>I have a (non-stateful) daily structured streaming ingestion job from json S3 file storage into a Delta table using Spark readStream. I would like to switch to Auto Loader to reduce file discovery time. My questions are:</P><P></P><UL><LI>Do I need to create a new streaming checkpoint? (I am guessing the answer to this question is yes)</LI><LI>Can I keep appending to the same Delta table (and table location), or do I need to create a new table as well?</LI></UL><P></P><P>Thank you!</P><P>Cosimo.</P> Mon, 18 Jul 2022 17:17:15 GMT Cosimo_F_ 2022-07-18T17:17:15Z https://community.databricks.com/t5/data-engineering/structured-streaming-migration-from-readstream-to-auto-loader/m-p/13129#M7848 <P>Hello!</P><P></P><P>I have a (non-stateful) daily structured streaming ingestion job from json S3 file storage into a Delta table using Spark readStream. I would like to switch to Auto Loader to reduce file discovery time. My questions are:</P><P></P><UL><LI>Do I need to create a new streaming checkpoint? (I am guessing the answer to this question is yes)</LI><LI>Can I keep appending to the same Delta table (and table location), or do I need to create a new table as well?</LI></UL><P></P><P>Thank you!</P><P>Cosimo.</P> Mon, 18 Jul 2022 17:17:15 GMT https://community.databricks.com/t5/data-engineering/structured-streaming-migration-from-readstream-to-auto-loader/m-p/13129#M7848 Cosimo_F_ 2022-07-18T17:17:15Z