- 6667 Views
- 14 replies
- 9 kudos
Suggestion Needed for a Orchestrator/Scheduler to schedule and execute Jobs in an automated way
Hello Friends,We have an application which extracts dat from various tables in Azure Databricks and we extract it to postgres tables (postgres installed on top of Azure VMs). After extraction we apply transformation on those datasets in postgres tabl...
- 6667 Views
- 14 replies
- 9 kudos
- 9 kudos
You can leverage Airflow, which provides a connector for databricks jobs API, or can use databricks workflow to orchestrate your jobs where you can define several tasks and set dependencies accordingly.
- 9 kudos
- 1014 Views
- 1 replies
- 0 kudos
How is the ETL process different than trigger once stream
I am little confused between what to use between structured stream(trigger once) and etl batch jobs, can I get help here on which basis i should make my decision.
- 1014 Views
- 1 replies
- 0 kudos
- 0 kudos
In Structured Streaming, triggers are used to specify how often a streaming query should produce results. A RunOnce trigger will fire only once and then will stop the query - effectively running it like a batch job.Now, If your source data is a strea...
- 0 kudos