GCS Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v1/in

KristiLogos — Wed, 12 Feb 2025 17:29:06 GMT

I’m running Databricks on Azure and trying to read a CSV file from Google Cloud Storage (GCS) bucket using Spark. However, despite configuring Spark with a Google service account key, I’m encountering the following error:

Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token

I’ve configured Spark with these settings to ensure it uses the service account for authentication following this document: https://docs.databricks.com/en/connect/storage/gcs.html

spark.conf.set("spark.hadoop.google.cloud.auth.service.account.enable", "true") spark.conf.set("spark.hadoop.fs.gs.auth.service.account.email", client_email) spark.conf.set("spark.hadoop.fs.gs.project.id", project_id) spark.conf.set("spark.hadoop.fs.gs.auth.service.account.private.key", private_key) spark.conf.set("spark.hadoop.fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem") spark.conf.set("spark.hadoop.fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS")spark.conf.set("spark.hadoop.fs.gs.auth.service.account.private.key.id", private_key_id)

Attempting to read the test csv file from my GCS bucket:

gcs_path = "gs://ddfsdfts/events/31dfsdfs4_2025_02_01_000000000000.csv"

df = spark.read.format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .load(gcs_path)

df.show()

The error happens when trying df.show()

I've seen a few other questions like this but no straight forward answers. Why is it trying to get to the metadata server token?

Re: GCS Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v

KristiLogos — Wed, 12 Feb 2025 18:36:07 GMT

I figured it out! I hope this helps someone else, I had to update the spark configuration in the cluster I was using, update with this:
https://docs.databricks.com/en/connect/storage/gcs.html#global-configuration

Re: GCS Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v

ShivangiB — Tue, 15 Apr 2025 08:05:35 GMT

Hey, @KristiLogos , can you please suggest in what format key was stored in gsa_private_key.Actually we are using key vault based scope

topic Re: GCS Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v in Data Engineering

GCS Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v1/in

Re: GCS Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v

Re: GCS Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v