topic Re: Getting the error 'No such file or directory', when trying to access the json file in Data Engineering

Getting the error 'No such file or directory', when trying to access the json file

Devarsh — Tue, 24 May 2022 05:47:58 GMT

I am trying to write in my google sheet through Databricks but when it comes to reading the json, file containing the credentials, I am getting the error that No such file or directory exists.

import gspread
 
 
gc = gspread.service_account(filename='dbfs:/FileStore/shared_uploads/abc@gmail.com/myapi_3115465_2ed05-3.json')
 
sh = gc.open_by_key('1********_g3kkhhA_9vMyp9piw')
 
worksheet = sh.sheet1
 
worksheet.update_cell(1,1,103032)

Getting the following error when I run the above code

FileNotFoundError: [Errno 2] No such file or directory: 'dbfs:/FileStore/shared_uploads/abc@gmail.com/myapi_3115465_2ed05-3.json'

Re: Getting the error 'No such file or directory', when trying to access the json file

-werners- — Wed, 25 May 2022 07:14:24 GMT

So open_by_key is a function of gspread, which is a python lib.

Pure python (not pyspark) will only read from local files,

see this topic also. It is about databricks-connect and pandas but the same principles apply.

Re: Getting the error 'No such file or directory', when trying to access the json file

Devarsh — Wed, 25 May 2022 09:23:27 GMT

Yes @Werner Stinckens, I tried different ways to read the json file from the path where it was stored but was unable to do so. Instead of this what I have done is that I have copied the content of the JSON file in a variable and passed that to the following function

credentials = {"type":"service_account","project_id":"myapi-.................}

gc = gspread.service_account_from_dict(credentials)

So now no need to read from any json file

Re: Getting the error 'No such file or directory', when trying to access the json file

Noopur_Nigam — Thu, 02 Jun 2022 03:57:45 GMT

Hi @Devarsh Shah The issue is not with json file but the location you are specifying while reading.As suggested by @Werner Stinckens please start using spark API to read the json file as below:

spark.read.format("json").load("testjson")

Please check the path you are providing in the read command if you still face the same issue.

https://docs.databricks.com/data/data-sources/read-json.html