Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-05-2018 01:22 AM
Just found a workaround for the issue with avro file read operation as it seems proper configuration for dfs.adls.oauth2.access.token.provider is not setup inside. If the ADL folder is mounted on databrick notebook , then it is working . Please try following steps
1. Mount adl folder
val configs = Map(
"dfs.adls.oauth2.access.token.provider.type" -> "ClientCredential",
"dfs.adls.oauth2.client.id" -> "XXX",
"dfs.adls.oauth2.credential" -> "YYY",
"dfs.adls.oauth2.refresh.url" -> "https://login.microsoftonline.com/ZZZ/oauth2/token",
"dfs.adls.oauth2.access.token.provider"->"org.apache.hadoop.fs.adls.oauth2.ConfCredentialBasedAccessTokenProvider")
dbutils.fs.mount(
source = "adl://XYZ.azuredatalakestore.net/myfolder/demo/",
mountPoint = "/mnt/mymount",
extraConfigs = configs)
2.Verify your file is visible on mount
dbutils.fs.ls("dbfs:/mnt/ashitabh3")import com.databricks.spark.avro._
spark.read.avro("dbfs:/mnt/mymount/mydata.avro").show
I can see the records now