Databricks Community

HaydenZhou · ‎12-10-2024

ala:570) at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:354) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.lang.NullPointerException at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:375) at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.connectUsingSASTokenProviderCredentials(AzureNativeFileSystemStore.java:1068) at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1173) ... 111 more shaded.databricks.org.apache.hadoop.fs.azure.AzureException: java.lang.NullPointerException at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1216) at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeF

the databricks runtime 16.0 read the azure blob storage file failed. from the logs we can see it call method.
`AzureNativeFileSystemStore.connectUsingSASTokenProviderCredentials`
But AzureNativeFileSystemStore did not have this method, this class only have `connectUsingSASCredentials` method

HaydenZhou · ‎12-10-2024

this is the test code removed sensetive credentials

package com.mycompany.hayden;

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;

import java.util.HashMap;

public class TestReadAzureCSV {
    public static void main(String[] args) {

        SparkSession spark = SparkSession.builder()
                .appName("ReadAzureBlobCSVWithSAS")
                .getOrCreate();


        String storageAccount = "abc";
        String containerName = "abc";
        String sasToken = "abc";
        String domain = "blob.core.windows.net"; //"z23.blob.storage.azure.net";

        spark.conf().set(String.format("fs.azure.account.auth.type.%s.%s",storageAccount, domain), "SAS");
        spark.conf().set(String.format("fs.azure.sas.%s.%s.%s", containerName,storageAccount, domain), sasToken);
        spark.sparkContext().hadoopConfiguration().set(String.format("fs.azure.sas.%s.%s.%s", containerName,storageAccount, domain), sasToken);


        String csvFilePath = "wasbs://xxx.csv";

        Dataset<Row> df = spark.read()
                .option("header", "true")
                .option("inferSchema", "true")
                .csv(csvFilePath);

        df.show(true);
    }

}

when I debug locally this is the method name entered.