😑Datafactory insists on Cluster compute BUT Databricks defaults to Serverless compute!

[Mizan1]

Hi all,

#Databricks

#Datafactory

I hope you are well.

I have bullet pointed my situation to allow other community members to easily diagnose my problem (I want to make your lives easy as well!).

I have also included screenshot of data pipeline in DataFactory to make your lives easier as well.

Background - Data Pipeline

In DataFactory, I am building a data pipeline with the following steps:

1. Using a Lookup block, the data is looked up from a source local SQL server
2. Using a DataLake block nested within a ForEach block, for each logic is used to iterate through multiple tables contained within the source SQL database and then dump those tables into a DataLake container called Bronze (I want to follow medallion methodology).
3. Using a DataBricks notebook block, the data is then transformed and then moved into a Silver datalake container.
   1. When configuring the DataBricks linked service, on the Select Cluster drop down, I selected 'Serverless'. This reflects on both DataBricks notebooks blocks.
4. Using a second DataBricks notebook block, the data is then transformed and then moved into a Gold datalake container.

Below is screenshot of pipeline diagram:

Issues - DataFactory wants Cluster compute for DataBricks notebook blocks BUT DataBricks doesn't allow me to create a Cluster 😑

When trying to publish my pipeline in DataFactory, I encounter the below error in Factory validation output:

When I try to return to DataBricks there is no option to create custom compute clusters. This was the case even before I even created the notebooks in the first place. On the left side bar, when I click on Compute, it leads me to SQL Warehouses instead. Screenshot below:

 

When creating the notebooks, I continued with the serverless option as that was the only way to process Python code for the transformations.

But now when trying to run the pipeline in datafactory, I am facing the error demanding me to make custom clusters.

Any help would be much appreciated!

Thanks

[balajij8]

The reason you are only seeing SQL Warehouses when clicking Compute is likely because the workspace admin has locked down interactive cluster creation. Check if you have the cluster creation entitlement. Your workspace admin can grant you the correct permissions.

You can follow below

• Job Cluster - You can use a "New job cluster" instead of selecting an existing interactive cluster or attempting to pass Serverless. When you add the notebook activity in ADF, setting it to create a new job cluster spins up ephemeral compute specifically for that run and terminates it immediately after. This mimics the cost-efficiency of serverless while remaining fully compatible with ADF's legacy notebook integration.
• Databricks Lakeflow Jobs - You can shift to Lakeflow jobs instead of having ADF orchestrate the individual notebooks. Databricks Jobs natively support Serverless compute. Once created, simply replace the ADF Notebook activities (bronze, silver notebooks etc) with a single ADF Databricks activity to trigger that Job. This routes the execution through the native Databricks plane while keeping the ADF pipeline clean.

[Mizan1 · Accepted solution]

Thank you for this. with your first solution, Datafactory returned an error saying that custom compute clusters is required for the notebook block to run.

As to the second solution, I didn't get round to trying that.

However, I found the solution to my issue where I had to recreate another workspace with Hybrid option enabled and not Serverless. This allowed me access to custom cluster creation.

Screenshot of workspace creation stage:

Screenshot of successful access to cluster compute:

I found solution from here: Re: Cannot click Compute tab - Databricks Community - 113970

Thank you for your time