I’m running Databricks on Azure and trying to read a CSV file from Google Cloud Storage (GCS) bucket using Spark. However, despite configuring Spark with a Google service account key, I’m encountering the following error:
Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token
I’ve configured Spark with these settings to ensure it uses the service account for authentication following this document: https://docs.databricks.com/en/connect/storage/gcs.html
I’ve configured Spark with these settings to ensure it uses the service account for authentication following this document: https://docs.databricks.com/en/connect/storage/gcs.html
spark.conf.set("spark.hadoop.google.cloud.auth.service.account.enable", "true") spark.conf.set("spark.hadoop.fs.gs.auth.service.account.email", client_email) spark.conf.set("spark.hadoop.fs.gs.project.id", project_id) spark.conf.set("spark.hadoop.fs.gs.auth.service.account.private.key", private_key) spark.conf.set("spark.hadoop.fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem") spark.conf.set("spark.hadoop.fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS")spark.conf.set("spark.hadoop.fs.gs.auth.service.account.private.key.id", private_key_id)
Attempting to read the test csv file from my GCS bucket:
gcs_path = "gs://ddfsdfts/events/31dfsdfs4_2025_02_01_000000000000.csv"
df = spark.read.format("csv") \
.option("header", "true") \
.option("inferSchema", "true") \
.load(gcs_path)
df.show()The error happens when trying df.show()
I've seen a few other questions like this but no straight forward answers. Why is it trying to get to the metadata server token?
