CDM connector for Spark can't connect to Azure storage account

Spark4speed — Thu, 22 Jun 2023 08:25:23 GMT

Hello,

I'm trying to use the CDM connector for Spark, but I can't connecto to the Azure storage account when using the connector.

I mounted a container of storage account with a SAS-token. When I'm trying to read CDM data from a (mounted) storage account, I get the following error:

java.lang.NoSuchMethodError: com.databricks.backend.daemon.data.client.adl.AdlGen2CredentialContextTokenProvider.getToken()Lshaded/databricks/v20180920_b33d810/org/apache/hadoop/fs/azurebfs/oauth2/AzureADToken;

I tried using a managed identity and a SAS token when using the connector, but without any luck.

Can I force the connector to use a SAS token?

I tried it like this and without the sas token:

df = (spark.read
    .format("com.microsoft.cdm")
    .option("storage", "<storage_account>")
    .option("manifestPath", "<path_to_manisfest_file>")
    .option("entity", "<entityname>")
    .option("sasToken", sas)
    .load()
)

What else can I try?

Re: CDM connector for Spark can't connect to Azure storage account

Anonymous — Fri, 23 Jun 2023 05:02:26 GMT

Hi @Martijn de Bruijn

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.

topic Re: CDM connector for Spark can't connect to Azure storage account in Data Engineering

CDM connector for Spark can't connect to Azure storage account

Re: CDM connector for Spark can't connect to Azure storage account