cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

How is the ETL process different than trigger once stream

User16826994223
Honored Contributor III

I am little confused between what to use between structured stream(trigger once) and etl batch jobs, can I get help here on which basis i should make my decision.

1 REPLY 1

sajith_appukutt
Honored Contributor II

In Structured Streaming, triggers are used to specify how often a streaming query should produce results. A RunOnce trigger will fire only once and then will stop the query - effectively running it like a batch job.

Now, If your source data is a streaming source, streaming and RunOnce is better than batch in general. Here is a blog post explaining this https://databricks.com/blog/2017/05/22/running-streaming-jobs-day-10x-cost-savings.html

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.