topic Re: Python open function is unable to detect the file in dbfs in Data Engineering

Python open function is unable to detect the file in dbfs

billaspiel — Thu, 25 Apr 2019 10:54:25 GMT

hi ,

Im a newbie learning spark using databricks , I did some investigation and searched if this questions was been asked earlier in community forum but unable to find anything so .

1. DBFS is unable to detect the file even though its present in it .

The issue happens only with below command

with open("dbfs:/FileStore/tables/data.txt") as f:

and not with

lines0 = sc.textFile("/FileStore/tables/data.txt"

Does this mean in databricks notebook we can't use python open function to open a file ?

Re: Python open function is unable to detect the file in dbfs

mathan_pillai — Thu, 25 Apr 2019 12:33:44 GMT

Hi @billaspiel

Can you please try by removing the colon ":" ? Here is a documentation on how to read/write the data

https://docs.databricks.com/user-guide/dbfs-databricks-file-system.html#access-dbfs-using-local-file-apis

If the issue still occurs, please share the complete statement and also the error message you are getting. Let us know how it goes.

Thanks

Re: Python open function is unable to detect the file in dbfs

billaspiel — Mon, 29 Apr 2019 07:11:22 GMT

Hi mathan,

Thanks for the information , in addition to removing colon':' added '/' as well before dbfs which resolved the issue .

working syntax

----------------------

with open("/dbfs/FileStore/tables/data.txt") as f:

Re: Python open function is unable to detect the file in dbfs

Dflo — Tue, 21 Mar 2023 21:24:04 GMT

I am having similar issues currently. I can read or access my storage account but when I attempted to read or access the container it told me path not found. I create the container and have full access as an owner.