https://community.databricks.com/t5/data-engineering/databricks-dlt-with-hive-metastore-and-adls-access-issues/m-p/80545#M36057 <P>We are currently working on Databricks DLT tables to transform data from bronze to silver. we are specifically instructed us not to use mount paths for accessing data from ADLS Gen 2. To comply, I configured storage credentials and created an external location, which allows us to access data from ADLS in any notebook using ABFSS URLs—this setup is working as expected.</P><P>Initially, we created DLT tables using the Unity Catalog method without any issues. However, we later learned that Unity Catalog cannot be used in production, prompting us to switch to the Hive Metastore. Unfortunately, while configuring DLT with Hive Metastore, we encountered an error asking us to configure the storage account key and token.</P><P>We are puzzled as to why our workspace can access data without issues, yet the DLT Hive Metastore approach is failing to do so. Given that we are strictly prohibited from creating mount points or configuring SAS tokens and storage keys, any suggestions or solutions would be greatly appreciated.<BR /><BR /><STRONG>Error -&nbsp;</STRONG>'</P><DIV><DIV><EM><SPAN class=""><SPAN class="">shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contracts.exceptions.KeyProviderException: Failure to initialize configuration for storage account &lt;storage-account-name&gt;.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.key</SPAN></SPAN></EM></DIV></DIV><DIV><DIV><EM><SPAN class=""><SPAN class="">shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException: Invalid configuration value detected for fs.azure.account.key&nbsp;</SPAN></SPAN></EM><SPAN>'</SPAN></DIV></DIV><P>&nbsp;</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="yvishal519_0-1721908544085.png" style="width: 373px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9893i567FB1D7A9CA4A42/image-dimensions/373x179/is-moderation-mode/true?v=v2" width="373" height="179" role="button" title="yvishal519_0-1721908544085.png" alt="yvishal519_0-1721908544085.png" /></span></P> Thu, 25 Jul 2024 12:02:48 GMT yvishal519 2024-07-25T12:02:48Z https://community.databricks.com/t5/data-engineering/databricks-dlt-with-hive-metastore-and-adls-access-issues/m-p/80545#M36057 <P>We are currently working on Databricks DLT tables to transform data from bronze to silver. we are specifically instructed us not to use mount paths for accessing data from ADLS Gen 2. To comply, I configured storage credentials and created an external location, which allows us to access data from ADLS in any notebook using ABFSS URLs—this setup is working as expected.</P><P>Initially, we created DLT tables using the Unity Catalog method without any issues. However, we later learned that Unity Catalog cannot be used in production, prompting us to switch to the Hive Metastore. Unfortunately, while configuring DLT with Hive Metastore, we encountered an error asking us to configure the storage account key and token.</P><P>We are puzzled as to why our workspace can access data without issues, yet the DLT Hive Metastore approach is failing to do so. Given that we are strictly prohibited from creating mount points or configuring SAS tokens and storage keys, any suggestions or solutions would be greatly appreciated.<BR /><BR /><STRONG>Error -&nbsp;</STRONG>'</P><DIV><DIV><EM><SPAN class=""><SPAN class="">shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contracts.exceptions.KeyProviderException: Failure to initialize configuration for storage account &lt;storage-account-name&gt;.dfs.core.windows.net: Invalid configuration value detected for fs.azure.account.key</SPAN></SPAN></EM></DIV></DIV><DIV><DIV><EM><SPAN class=""><SPAN class="">shaded.databricks.azurebfs.org.apache.hadoop.fs.azurebfs.contracts.exceptions.InvalidConfigurationValueException: Invalid configuration value detected for fs.azure.account.key&nbsp;</SPAN></SPAN></EM><SPAN>'</SPAN></DIV></DIV><P>&nbsp;</P><P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="yvishal519_0-1721908544085.png" style="width: 373px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/9893i567FB1D7A9CA4A42/image-dimensions/373x179/is-moderation-mode/true?v=v2" width="373" height="179" role="button" title="yvishal519_0-1721908544085.png" alt="yvishal519_0-1721908544085.png" /></span></P> Thu, 25 Jul 2024 12:02:48 GMT https://community.databricks.com/t5/data-engineering/databricks-dlt-with-hive-metastore-and-adls-access-issues/m-p/80545#M36057 yvishal519 2024-07-25T12:02:48Z https://community.databricks.com/t5/data-engineering/databricks-dlt-with-hive-metastore-and-adls-access-issues/m-p/80566#M36064 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/108898">@yvishal519</a>&nbsp;,<BR /><BR />Since you're using hive metastore you have no other option than mount points. Storage credentials and external locations are only supported in Unity Catalog</P> Thu, 25 Jul 2024 12:42:26 GMT https://community.databricks.com/t5/data-engineering/databricks-dlt-with-hive-metastore-and-adls-access-issues/m-p/80566#M36064 szymon_dybczak 2024-07-25T12:42:26Z https://community.databricks.com/t5/data-engineering/databricks-dlt-with-hive-metastore-and-adls-access-issues/m-p/80570#M36065 <P>Thank you for your quick responses! <EM>Could you please share any documentation that mentions that storage credentials and external locations are only supported in Unity Catalog?</EM></P> Thu, 25 Jul 2024 12:56:26 GMT https://community.databricks.com/t5/data-engineering/databricks-dlt-with-hive-metastore-and-adls-access-issues/m-p/80570#M36065 yvishal519 2024-07-25T12:56:26Z