topic Re: Enable credential passthrough Option is not available in new UI for Job Cluster in Data Engineering

Enable credential passthrough Option is not available in new UI for Job Cluster

Deepak_Kandpal — Tue, 13 Sep 2022 09:42:13 GMT

Hi All,

I am trying to add new workflow which require to use credential passthrough, but when I am trying to create new Job Cluster from Workflow -> Jobs -> My Job, the option of Enable credential passthrough is not available.

Is there any other way to enable this.

Regards,

Deepak

Re: Enable credential passthrough Option is not available in new UI for Job Cluster

Deepak_Kandpal — Wed, 14 Sep 2022 06:23:46 GMT

Hi Debayan,

Thankyou for your response. I am working on a project where it require to read and process Excel file and I am using Maven Spark Excel library to read the excel file, and it is successfully reading on All purpose cluster with credential passthrough option. However now we need to put this notebook into schedule using workflows and due to lack of credential passthrough option for Job Cluster we are getting below error.

Error: Failure to initialize configuration Invalid configuration value detected for fs.azure.account.key

Is there a workaround to run this notebook with Job Cluster? As we want to run all workflows using a service account.

Regards,

Deepak

Re: Enable credential passthrough Option is not available in new UI for Job Cluster

Rostislaw — Thu, 15 Sep 2022 19:25:54 GMT

assuming your Excel file is located on ADLS you can add a service principal to the cluster configuration. see: https://docs.microsoft.com/en-us/azure/databricks/data/data-sources/azure/azure-storage#--access-azure-data-lake-storage-gen2-or-blob-storage-using-oauth-20-with-an-azure-service-principal

With Unity Catalog you could probably also use authorization based on the external path.

Re: Enable credential passthrough Option is not available in new UI for Job Cluster

Debayan — Tue, 13 Sep 2022 14:35:10 GMT

Hi Deepak, Thanks for reaching out to community.databricks.com.

Job cluster is not having option to enable credential passthrough.