topic Re: Change cluster mid-pipeline in Data Engineering

Change cluster mid-pipeline

niels — Mon, 02 May 2022 09:08:14 GMT

I have a notebook functioning as a pipeline, where multiple notebooks are chained together.

The issue I'm facing is that some of the notebooks are spark-optimized, others aren't, and what I want is to use 1 cluster for the former and another for the latter. However, this would mean changing clusters halfway through the pipeline notebook. Is that possible? And if so, how?

Re: Change cluster mid-pipeline

Hubert-Dudek — Mon, 02 May 2022 09:11:17 GMT

In such a case, orchestrating those jobs using Azure Data Factory is highly recommended.

Re: Change cluster mid-pipeline

Prabakar — Mon, 02 May 2022 21:13:25 GMT

Yes, you can achieve this by setting two different job clusters. In the screenshot, you can see I have used 2 job clusters PipelineTest and pipelinetest2. You can refer the doc https://docs.databricks.com/jobs.html#cluster-config-tips

Re: Change cluster mid-pipeline

niels — Tue, 26 Jul 2022 08:28:41 GMT

Hi Kaniz, sorry for the incredibly late reply. My notifications for responses ended up in my spam folder!

I ended up using ADF, but tried @Prabakar Ammeappin 's solution and that worked too!