cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

GCS Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v1/in

KristiLogos
Contributor

Iā€™m running Databricks on Azure and trying to read a CSV file from Google Cloud Storage (GCS) bucket using Spark. However, despite configuring Spark with a Google service account key, Iā€™m encountering the following error:

Error getting access token from metadata server at: http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token

Iā€™ve configured Spark with these settings to ensure it uses the service account for authentication following this document: https://docs.databricks.com/en/connect/storage/gcs.html

Iā€™ve configured Spark with these settings to ensure it uses the service account for authentication following this document: https://docs.databricks.com/en/connect/storage/gcs.html

spark.conf.set("spark.hadoop.google.cloud.auth.service.account.enable", "true") spark.conf.set("spark.hadoop.fs.gs.auth.service.account.email", client_email) spark.conf.set("spark.hadoop.fs.gs.project.id", project_id) spark.conf.set("spark.hadoop.fs.gs.auth.service.account.private.key", private_key) spark.conf.set("spark.hadoop.fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem") spark.conf.set("spark.hadoop.fs.AbstractFileSystem.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFS")spark.conf.set("spark.hadoop.fs.gs.auth.service.account.private.key.id", private_key_id)

Attempting to read the test csv file from my GCS bucket:

gcs_path = "gs://ddfsdfts/events/31dfsdfs4_2025_02_01_000000000000.csv"

df = spark.read.format("csv") \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .load(gcs_path)

df.show()

The error happens when trying df.show()

I've seen a few other questions like this but no straight forward answers. Why is it trying to get to the metadata server token?

1 ACCEPTED SOLUTION

Accepted Solutions

KristiLogos
Contributor

I figured it out! I hope this helps someone else, I had to update the spark configuration in the cluster I was using, update with this:
https://docs.databricks.com/en/connect/storage/gcs.html#global-configuration

View solution in original post

1 REPLY 1

KristiLogos
Contributor

I figured it out! I hope this helps someone else, I had to update the spark configuration in the cluster I was using, update with this:
https://docs.databricks.com/en/connect/storage/gcs.html#global-configuration

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonā€™t want to miss the chance to attend and share knowledge.

If there isnā€™t a group near you, start one and help create a community that brings people together.

Request a New Group