Enable credential passthrough Option is not available in new UI for Job Cluster

[Deepak_Kandpal]

Hi All,

I am trying to add new workflow which require to use credential passthrough, but when I am trying to create new Job Cluster from Workflow -> Jobs -> My Job, the option of Enable credential passthrough is not available.

Is there any other way to enable this.

 

Regards,

Deepak

[Debayan]

Hi Deepak, Thanks for reaching out to community.databricks.com.

Job cluster is not having option to enable credential passthrough.

[Deepak_Kandpal]

Hi Debayan,

Thankyou for your response. I am working on a project where it require to read and process Excel file and I am using Maven Spark Excel library to read the excel file, and it is successfully reading on All purpose cluster with credential passthrough option. However now we need to put this notebook into schedule using workflows and due to lack of credential passthrough option for Job Cluster we are getting below error.

Error: Failure to initialize configuration Invalid configuration value detected for fs.azure.account.key

Is there a workaround to run this notebook with Job Cluster? As we want to run all workflows using a service account.

Regards,

Deepak

[Rostislaw · Accepted solution]

assuming your Excel file is located on ADLS you can add a service principal to the cluster configuration. see: https://docs.microsoft.com/en-us/azure/databricks/data/data-sources/azure/azure-storage#--access-azu...

With Unity Catalog you could probably also use authorization based on the external path.