Databricks

Spark4speed · ‎06-22-2023

Hello,

I'm trying to use the CDM connector for Spark, but I can't connecto to the Azure storage account when using the connector.

I mounted a container of storage account with a SAS-token. When I'm trying to read CDM data from a (mounted) storage account, I get the following error:

java.lang.NoSuchMethodError: com.databricks.backend.daemon.data.client.adl.AdlGen2CredentialContextTokenProvider.getToken()Lshaded/databricks/v20180920_b33d810/org/apache/hadoop/fs/azurebfs/oauth2/AzureADToken;

I tried using a managed identity and a SAS token when using the connector, but without any luck.

Can I force the connector to use a SAS token?

I tried it like this and without the sas token:

df = (spark.read
    .format("com.microsoft.cdm")
    .option("storage", "<storage_account>")
    .option("manifestPath", "<path_to_manisfest_file>")
    .option("entity", "<entityname>")
    .option("sasToken", sas)
    .load()
)

What else can I try?