How is the ETL process different than trigger once stream

User16826994223
Databricks Employee
Databricks Employee

‎06-17-2021 12:54 AM

I am little confused between what to use between structured stream(trigger once) and etl batch jobs, can I get help here on which basis i should make my decision.

Labels:
0 Kudos

sajith_appukutt
Databricks Employee
Databricks Employee

‎06-22-2021 10:39 PM

In Structured Streaming, triggers are used to specify how often a streaming query should produce results. A RunOnce trigger will fire only once and then will stop the query - effectively running it like a batch job.

Now, If your source data is a streaming source, streaming and RunOnce is better than batch in general. Here is a blog post explaining this https://databricks.com/blog/2017/05/22/running-streaming-jobs-day-10x-cost-savings.html

0 Kudos