cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Are Jobs not supported on cluster with Azure Data Lake Storage credential passthrough enabled cluster?

nancy_g
New Contributor III
1 ACCEPTED SOLUTION

Accepted Solutions

User16764241763
Honored Contributor

Hello @Nancy Guptaโ€‹ 

Jobs passthrough is currently in private preview. Could you please reach out your Databricks Account representatives to help you with onboarding?

Or Please raise a support case with us, so we can assist you.

View solution in original post

4 REPLIES 4

nancy_g
New Contributor III

I found a documentation which mentions that Jobs are not supported, https://github.com/hurtn/datalake-ADLS-access-patterns-with-Databricks/blob/master/readme.md#Pattern...

Wanted to confirm if this holds true and this credential passthrough support is only for notebook?

User16764241763
Honored Contributor

Hello @Nancy Guptaโ€‹ 

Jobs passthrough is currently in private preview. Could you please reach out your Databricks Account representatives to help you with onboarding?

Or Please raise a support case with us, so we can assist you.

Juan
New Contributor II

This is utterly ridiculous that jobs are not supported with this as a standard, so am I left to execute jobs manually?

Rostislaw
New Contributor III

Right now the feature seems to be public available. It is possible to schedule jobs with ADLS passthough enabled and do not have to provide service principal credentials.

However I ask myself how that works behind the scenses. When working interactively with notebooks the "passthrough" refers to the fact that the user is logged in and already available access_token can be used for access ADLS. Job clusters runs unattended and there is no access_token available...

Does anybody know which OAuth2 flow Databricks use in order to get the access_token of the job owner?

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now