Python open function is unable to detect the file in dbfs

billaspiel
New Contributor II

‎04-25-2019 03:54 AM

hi ,

Im a newbie learning spark using databricks , I did some investigation and searched if this questions was been asked earlier in community forum but unable to find anything so .

1. DBFS is unable to detect the file even though its present in it .

The issue happens only with below command

with open("dbfs:/FileStore/tables/data.txt") as f:

and not with

lines0 = sc.textFile("/FileStore/tables/data.txt"

Does this mean in databricks notebook we can't use python open function to open a file ?

Labels:
0 Kudos

mathan_pillai
Databricks Employee
Databricks Employee

‎04-25-2019 05:33 AM

Hi @billaspiel

Can you please try by removing the colon ":" ? Here is a documentation on how to read/write the data

https://docs.databricks.com/user-guide/dbfs-databricks-file-system.html#access-dbfs-using-local-file...

If the issue still occurs, please share the complete statement and also the error message you are getting. Let us know how it goes.

Thanks

View solution in original post

0 Kudos

billaspiel
New Contributor II
In response to mathan_pillai

‎04-29-2019 12:11 AM

Hi mathan,

Thanks for the information , in addition to removing colon':' added '/' as well before dbfs which resolved the issue .

working syntax

----------------------

with open("/dbfs/FileStore/tables/data.txt") as f:

0 Kudos

Dflo
New Contributor II

‎03-21-2023 02:24 PM

I am having similar issues currently. I can read or access my storage account but when I attempted to read or access the container it told me path not found. I create the container and have full access as an owner.

0 Kudos