https://community.databricks.com/t5/data-engineering/wal-for-structured-streaming/m-p/63727#M32334 <P>Hi,&nbsp;</P><P>I cannot find deep-dive on this from latest links. So far the understanding is:</P><P>Previously SS (structured streaming) copies and caches the data in WAL. After a version, with retrieve less, SS doesn't copy the data to WAL any more, and only stores "offset", and WAL is not being used any more and only depends on checkpoint. Is this understanding right?&nbsp;</P> Thu, 14 Mar 2024 18:17:43 GMT MikeGo 2024-03-14T18:17:43Z https://community.databricks.com/t5/data-engineering/wal-for-structured-streaming/m-p/63727#M32334 <P>Hi,&nbsp;</P><P>I cannot find deep-dive on this from latest links. So far the understanding is:</P><P>Previously SS (structured streaming) copies and caches the data in WAL. After a version, with retrieve less, SS doesn't copy the data to WAL any more, and only stores "offset", and WAL is not being used any more and only depends on checkpoint. Is this understanding right?&nbsp;</P> Thu, 14 Mar 2024 18:17:43 GMT https://community.databricks.com/t5/data-engineering/wal-for-structured-streaming/m-p/63727#M32334 MikeGo 2024-03-14T18:17:43Z https://community.databricks.com/t5/data-engineering/wal-for-structured-streaming/m-p/63840#M32378 <P>Thanks Kaniz.&nbsp;</P><P>Theoretically even if without WAL, everything can be recovered from checkpoint right? Does the WAL exist only for perf reasons? E.g. for a micro batch, Spark might run multiple batches inside the microbatch and WAL is used to record the state of each micro micro-batch?&nbsp;</P> Fri, 15 Mar 2024 17:05:23 GMT https://community.databricks.com/t5/data-engineering/wal-for-structured-streaming/m-p/63840#M32378 MikeGo 2024-03-15T17:05:23Z