topic Re: Mounted Azure Storage shows mount.err inside folder while reading from Azure Databricks in Data Engineering

Mounted Azure Storage shows mount.err inside folder while reading from Azure Databricks

vichus1995 — Wed, 03 May 2023 04:01:37 GMT

I'm using Azure Databricks notebook to read a excel file from a folder inside a mounted Azure blob storage.

The mounted excel location is like : "/mnt/2023-project/dashboard/ext/Marks.xlsx". 2023-project is the mount point and dashboard is the name of the container.

When I do a dbutils.fs.ls I can see all the files inside the ext folder. There are lot of os functions being used in the code as it was developed on a different environment.

When I do a os.listdir on ext folder, I get a error No such file or directory. When I do a os.listdir on dashboard container I get mount.err as the output. While reading the excel file using pandas or openpyxl I get a error No such file or directory.

I have tried with and without using /dbfs at the beginning of the mount point.

I'm using DBR 12.1 (includes Apache Spark 3.3.1, Scala 2.12). I mounted the azure storage using the credential pass through method.

configs = {

'fs.azure.account.auth.type': 'CustomAccessToken',

'fs.azure.account.custom.token.provider.class': spark.conf.get('spark.databricks.passthrough.adls.gen2.tokenProviderClassName')

}

Please help on this. I'm relatively new to Databricks.

Re: Mounted Azure Storage shows mount.err inside folder while reading from Azure Databricks

-werners- — Wed, 03 May 2023 08:48:03 GMT

If you want to learn using databricks, I would first read the file into a spark dataframe using spark.read

This results in a spark dataframe. From that moment on you can decide going further using pyspark or covert the dataframe to a pandas dataframe.

This link has some info:

https://learn.microsoft.com/en-us/azure/synapse-analytics/quickstart-read-from-gen2-to-pandas-dataframe

Re: Mounted Azure Storage shows mount.err inside folder while reading from Azure Databricks

Anonymous — Fri, 19 May 2023 07:44:52 GMT

Hi @vichus1995

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!