topic Python spark.read.text Path does not exist in Data Engineering

Python spark.read.text Path does not exist

rba76 — Mon, 21 Oct 2019 08:42:56 GMT

Dear all,

I want to read files with python from a storage account. I followed this instruction https://docs.microsoft.com/en-us/azure/azure-databricks/store-secrets-azure-key-vault.

This is my python code:

dbutils.fs.mount(source = "wasbs://container1@datalake1.blob.core.windows.net", mount_point = "/mnt/container1", extra_configs = {"fs.azure.sas.container1.datalake1.blob.core.windows.net":dbutils.secrets.get(scope = "secretscope", key = "secret")})

df = spark.read.text("mnt/container1/helloworld.txt")

df.show()

The mount part runs succesfull, but it is not possible to read the file which is located in the container. I got this error "AnalysisException: 'Path does not exist: dbfs:/mnt/container1/helloworld.txt;'".

Is this a security reason?

Any help would be appreciated!!! Thank you very much!!

best regards,

Rudolf

Re: Python spark.read.text Path does not exist

shyam_9 — Tue, 22 Oct 2019 07:19:44 GMT

Hi @rba76,

Please follow as below,

https://docs.azuredatabricks.net/data/data-sources/azure/azure-storage.html

Re: Python spark.read.text Path does not exist

PRADEEPCHEEKATL — Fri, 25 Oct 2019 10:30:52 GMT

@rba76 Make sure helloworld.txt file exists in the container1 folder

I'm able to view the text file using the same commands as follows:

Mount Blob Storage:

dbutils.fs.mount(
  source = "wasbs://sampledata@azure.blob.core.windows.net/Azure",
  mount_point = "/mnt/chepra",
  extra_configs = {"fs.azure.sas.sampledata.azure.blob.core.windows.net":dbutils.secrets.get(scope = "azurestorage", key = "azurestoragekey")})

Access files in your container as if they were local files, for example:

df = spark.read.text("/mnt/chepra/systeminfo.txt")
df.show()

Hope this helps.