topic Is it possible to write single Spark stream to 2 different Delta tables? Recommendations around that? in Data Engineering https://community.databricks.com/t5/data-engineering/is-it-possible-to-write-single-spark-stream-to-2-different-delta/m-p/23111#M15919 Thu, 17 Jun 2021 18:34:08 GMT Anonymous 2021-06-17T18:34:08Z Is it possible to write single Spark stream to 2 different Delta tables? Recommendations around that? https://community.databricks.com/t5/data-engineering/is-it-possible-to-write-single-spark-stream-to-2-different-delta/m-p/23111#M15919 Thu, 17 Jun 2021 18:34:08 GMT https://community.databricks.com/t5/data-engineering/is-it-possible-to-write-single-spark-stream-to-2-different-delta/m-p/23111#M15919 Anonymous 2021-06-17T18:34:08Z Re: Is it possible to write single Spark stream to 2 different Delta tables? Recommendations around that? https://community.databricks.com/t5/data-engineering/is-it-possible-to-write-single-spark-stream-to-2-different-delta/m-p/23112#M15920 <P>In this scenario, the best option would be to have a single readStream reading a source delta table. Since checkpoint logs are controlled when writing to delta tables you would be able to maintain separate logs for each of your writeStreams. </P><P></P><P>I would recommend storing your checkpoint file in a `_checkpoint` sub-directory in your delta table directory. The "_" will allow spark to ignore it when reading the table and you are able to maintain those checkpoints with the appropriate tables. </P> Thu, 17 Jun 2021 19:36:20 GMT https://community.databricks.com/t5/data-engineering/is-it-possible-to-write-single-spark-stream-to-2-different-delta/m-p/23112#M15920 Ryan_Chynoweth 2021-06-17T19:36:20Z