Shutting down a job cluster, when streaming is over

[pawelmitrus]

Hi,

As for now we already know that our application will be running 24/7 streaming constantly incoming data. The stream pipeline is very basic, however as for now it's enough to run this pipeline 1x per day (to save the costs of constantly running cluster). I'd like to write the code already to stream the data, instead of batch. I've been wondering how to detect that for a given time all new files were handled and streamed, so the cluster can shut down. When the job starts the next day, it will pick up only new files.

Could you please advise anything on that?

Thanks,

Pawel

[Shasidhar_ES · Accepted solution]

Use .trigger(once=True) or .trigger(availableNow=True) option which will pick only the new files https://docs.databricks.com/structured-streaming/triggers.html#configuring-incremental-batch-process...