DBR 16.0 spark read Azure Blob file Failed.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-10-2024 07:46 PM - edited 12-10-2024 07:49 PM
ala:570) at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:354) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: java.lang.NullPointerException at java.base/java.lang.Class.forName0(Native Method) at java.base/java.lang.Class.forName(Class.java:375) at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.connectUsingSASTokenProviderCredentials(AzureNativeFileSystemStore.java:1068) at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1173) ... 111 more shaded.databricks.org.apache.hadoop.fs.azure.AzureException: java.lang.NullPointerException at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.createAzureStorageSession(AzureNativeFileSystemStore.java:1216) at shaded.databricks.org.apache.hadoop.fs.azure.AzureNativeF
the databricks runtime 16.0 read the azure blob storage file failed. from the logs we can see it call method.
`AzureNativeFileSystemStore.connectUsingSASTokenProviderCredentials`
But AzureNativeFileSystemStore did not have this method, this class only have `connectUsingSASCredentials` method
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-10-2024 07:48 PM - edited 12-10-2024 07:51 PM
this is the test code removed sensetive credentials
package com.mycompany.hayden;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.SparkSession;
import java.util.HashMap;
public class TestReadAzureCSV {
public static void main(String[] args) {
SparkSession spark = SparkSession.builder()
.appName("ReadAzureBlobCSVWithSAS")
.getOrCreate();
String storageAccount = "abc";
String containerName = "abc";
String sasToken = "abc";
String domain = "blob.core.windows.net"; //"z23.blob.storage.azure.net";
spark.conf().set(String.format("fs.azure.account.auth.type.%s.%s",storageAccount, domain), "SAS");
spark.conf().set(String.format("fs.azure.sas.%s.%s.%s", containerName,storageAccount, domain), sasToken);
spark.sparkContext().hadoopConfiguration().set(String.format("fs.azure.sas.%s.%s.%s", containerName,storageAccount, domain), sasToken);
String csvFilePath = "wasbs://xxx.csv";
Dataset<Row> df = spark.read()
.option("header", "true")
.option("inferSchema", "true")
.csv(csvFilePath);
df.show(true);
}
}when I debug locally this is the method name entered.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-12-2024 05:21 PM
I had find the solustion of this problem. Close this Post.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-13-2024 01:29 AM
Hi @HaydenZhou ,
Could you share the solution with community, so if one encounter similar issue could use similiar approach?