How to access ADLS Gen2 hdfs from a databricks cluster which has credential passthrough enabled?
Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
07-21-2023 02:30 AM
When executing through a Databricks cluster with credential passthrough enabled, I wish to obtain supplementary file attributes in ADLS, such as the file's last modified time, which are currently unavailable in the databricks dbutils.fs.ls function.
We can use Azure DataLake service client library to access ADLS File system, but it's doesn't support databricks native credential passthrough authentication type.
Above scenario goes with the presumption that user already has access to ADLS GEN2 through AD Group role assignment.