How is the ETL process different than trigger once stream

User16826994223 — Thu, 17 Jun 2021 07:54:41 GMT

I am little confused between what to use between structured stream(trigger once) and etl batch jobs, can I get help here on which basis i should make my decision.

Re: How is the ETL process different than trigger once stream

sajith_appukutt — Wed, 23 Jun 2021 05:39:50 GMT

In Structured Streaming, triggers are used to specify how often a streaming query should produce results. A RunOnce trigger will fire only once and then will stop the query - effectively running it like a batch job.

Now, If your source data is a streaming source, streaming and RunOnce is better than batch in general. Here is a blog post explaining this https://databricks.com/blog/2017/05/22/running-streaming-jobs-day-10x-cost-savings.html

topic How is the ETL process different than trigger once stream in Data Engineering

How is the ETL process different than trigger once stream

Re: How is the ETL process different than trigger once stream