topic Re: Should/Can I use spark streaming for Batch workloads? in Data Engineering

Should/Can I use spark streaming for Batch workloads?

User16783855534 — Wed, 23 Jun 2021 19:50:12 GMT

Its preferable to use spark streaming (with Delta) for batch workloads rather then regular batch. With the trigger.once trigger whenever the streaming job is started it will process whatever is available in the source (kafka/kinesis/File System) and keep track of the progress in the streaming checkpoint location. So after it succeeds when you run the job again it will leverage checkpoint to know where to start from.

Re: Should/Can I use spark streaming for Batch workloads?

User16826994223 — Thu, 24 Jun 2021 01:05:03 GMT

Re: Should/Can I use spark streaming for Batch workloads?

brickster_2018 — Thu, 24 Jun 2021 13:52:51 GMT

The streaming checkpoint mechanism is independent of the Trigger type. The way checkpoint works are it creates an offset file when processing the batch and once the batch is completed it creates a commit file for that batch in the checkpoint directory. Irrespective of the Trigger type, whenever a batch starts it will first reconcile the offset and checkpoint directory to identify where it has to resume.

These files are human-readable and can be seen in the checkpoint directory.