topic Re: DBR 16.0 spark read Azure Blob file Failed. in Data Engineering

DBR 16.0 spark read Azure Blob file Failed.

HaydenZhou — Wed, 11 Dec 2024 03:49:01 GMT

ala:570) at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:354) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.lang.NullPointerException at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:375) at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.connectUsingSASTokenProviderCredentials(AzureNativeFileSystemStore.java:1068) at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1173) ... 111 more shaded.databricks.org.apache.hadoop.fs.azure.AzureException: java.lang.NullPointerException at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1216) at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeF

the databricks runtime 16.0 read the azure blob storage file failed. from the logs we can see it call method.
`AzureNativeFileSystemStore.connectUsingSASTokenProviderCredentials`
But AzureNativeFileSystemStore did not have this method, this class only have `connectUsingSASCredentials` method

Re: DBR 16.0 spark read Azure Blob file Failed.

HaydenZhou — Wed, 11 Dec 2024 03:51:49 GMT

this is the test code removed sensetive credentials

package com.mycompany.hayden; import org.apache.spark.sql.Dataset; import org.apache.spark.sql.Row; import org.apache.spark.sql.SparkSession; import java.util.HashMap; public class TestReadAzureCSV { public static void main(String[] args) { SparkSession spark = SparkSession.builder() .appName("ReadAzureBlobCSVWithSAS") .getOrCreate(); String storageAccount = "abc"; String containerName = "abc"; String sasToken = "abc"; String domain = "blob.core.windows.net"; //"z23.blob.storage.azure.net"; spark.conf().set(String.format("fs.azure.account.auth.type.%s.%s",storageAccount, domain), "SAS"); spark.conf().set(String.format("fs.azure.sas.%s.%s.%s", containerName,storageAccount, domain), sasToken); spark.sparkContext().hadoopConfiguration().set(String.format("fs.azure.sas.%s.%s.%s", containerName,storageAccount, domain), sasToken); String csvFilePath = "wasbs://xxx.csv"; Dataset<Row> df = spark.read() .option("header", "true") .option("inferSchema", "true") .csv(csvFilePath); df.show(true); } }

when I debug locally this is the method name entered.

Re: DBR 16.0 spark read Azure Blob file Failed.

HaydenZhou — Fri, 13 Dec 2024 01:21:07 GMT

I had find the solustion of this problem. Close this Post.

Re: DBR 16.0 spark read Azure Blob file Failed.

szymon_dybczak — Fri, 13 Dec 2024 09:29:51 GMT

Hi @HaydenZhou ,

Could you share the solution with community, so if one encounter similar issue could use similiar approach?