Can not read data from GCS

shihs — Wed, 27 Sep 2023 20:02:38 GMT

I am trying to use Databricks to read data on Google Cloud Storage (GCS) with Databricks on Google Cloud. I followed the steps from https://docs.gcp.databricks.com/storage/gcs.html.

I have tried Access GCS buckets using Google Cloud service accounts on clusters, but I still couldn't read the file on GCS with the code below

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("test").getOrCreate()

df = spark.read.format("csv").load("gs://mybucket/test.csv")

The error message I got

```
"xxx@xxx.iam.gserviceaccount.com does not have storage.objects.get access to the Google Cloud Storage object. Permission 'storage.objects.get' denied on resource (or it may not exist).",
```

I also tried Access a GCS bucket directly with a Google Cloud service account key. I stucked in Step 4 & 5. Since step 5 uses `{{secrets/scope/gsa_private_key}}` and `{{secrets/scope/gsa_private_key_id}}` to get thegsa_private_key and gsa_private_key_id. I am not quite sure where should I do the step 4? I think it doesn't make to do it on local computer, however, it is also weird to do it on the cluster terminal.

Please help me solve this problem. Thanks in advance!

topic Can not read data from GCS in Get Started Discussions

Can not read data from GCS