mount a Azure DL in Databricks

Krish123 — Sat, 26 Mar 2022 03:19:19 GMT

Hello Team,

I am quite new to Databricks and I am learning PySpark and Databricks.

I am trying to mount a DL Gen2 in Databricks, as part of that I had created app registration, added DL into app registration permissions, created a secret and also added the clientid, secret and tenantid in azure vault. I also created the scope too, but I am getting below error:

Please advice, what's the mistake.

ExecutionError: An error occurred while calling o376.mount.

: java.lang.NullPointerException: authEndpoint

at shaded.databricks.v20180920_b33d810.com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)

at shaded.databricks.v20180920_b33d810.org.apache.hadoop.fs.azurebfs.oauth2.AzureADAuthenticator.getTokenUsingClientCreds(AzureADAuthenticator.java:84)

at com.databricks.backend.daemon.dbutils.DBUtilsCore.verifyAzureOAuth(DBUtilsCore.scala:751)

at com.databricks.backend.daemon.dbutils.DBUtilsCore.verifyAzureFileSystem(DBUtilsCore.scala:762)

at com.databricks.backend.daemon.dbutils.DBUtilsCore.mount(DBUtilsCore.scala:720)

at sun.reflect.GeneratedMethodAccessor291.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)

at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:380)

at py4j.Gateway.invoke(Gateway.java:295)

at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)

at py4j.commands.CallCommand.execute(CallCommand.java:79)

at py4j.GatewayConnection.run(GatewayConnection.java:251)

at java.lang.Thread.run(Thread.java:748)

Script I ran, to mount:

applicationId=dbutils.secrets.get(scope="testscope",key="clientid")

authenticationKey=dbutils.secrets.get(scope="testscope",key="clientsecret")

tenandId=dbutils.secrets.get(scope="testscope",key="tenantid")

endpoint="https://login.microsoftonline.com/"+tenandId+"/oauth2/token"

source="abfss://"+adlsContainerName+"@"+adlsAccountName+".dfs.core.windows.net/"+adlsFolderName

# connecting using service principle oAuth

configs = {"fs.azure.account.auth.type": "OAuth",

"fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",

"fs.azure.account.oauth2.client.id": applicationId,

"fs.azure.account.oauth2.client.secret": authenticationKey,

"fs.azure.account.oauth2.client.endoint": "https://login.microsoftonline.com/"+tenandId+"/oauth2/token"}

## mounting the ADLS storage to Databricks. Mount only if the directory is not already mounted

dbutils.fs.mount(

source = source,

mount_point = "/mnt/adls",

extra_configs = configs)

topic mount a Azure DL in Databricks in Data Engineering

mount a Azure DL in Databricks