Hi all,

#Databricks

#Datafactory

I hope you are well.

I have bullet pointed my situation to allow other community members to easily diagnose my problem (I want to make your lives easy as well!).

I have also included screenshot of data pipeline in DataFactory to make your lives easier as well.

Background - Data Pipeline

In DataFactory, I am building a data pipeline with the following steps:

1. Using a Lookup block, the data is looked up from a source local SQL server
2. Using a DataLake block nested within a ForEach block, for each logic is used to iterate through multiple tables contained within the source SQL database and then dump those tables into a DataLake container called Bronze (I want to follow medallion methodology).
3. Using a DataBricks notebook block, the data is then transformed and then moved into a Silver datalake container.
   1. When configuring the DataBricks linked service, on the Select Cluster drop down, I selected 'Serverless'. This reflects on both DataBricks notebooks blocks.
4. Using a second DataBricks notebook block, the data is then transformed and then moved into a Gold datalake container.

Below is screenshot of pipeline diagram:

Issues - DataFactory wants Cluster compute for DataBricks notebook blocks BUT DataBricks doesn't allow me to create a Cluster 😑

When trying to publish my pipeline in DataFactory, I encounter the below error in Factory validation output:

When I try to return to DataBricks there is no option to create custom compute clusters. This was the case even before I even created the notebooks in the first place. On the left side bar, when I click on Compute, it leads me to SQL Warehouses instead. Screenshot below:

When creating the notebooks, I continued with the serverless option as that was the only way to process Python code for the transformations.

But now when trying to run the pipeline in datafactory, I am facing the error demanding me to make custom clusters.

Any help would be much appreciated!

Thanks