Not able to connect to GCP Secret Manager except when using "No isolation shared" Cluster

[yumnus]

Hey everyone,

We’re trying to access secrets stored in GCP Secret Manager using its Python package from Databricks on GCP. However, we can only reach the Secret Manager when using "No Isolation Shared" clusters, which is not an option for us. Currently, we haven’t found any alternative solutions.

Has anyone encountered this issue or found a workaround?

The error message indicates that Google’s metadata server is unreachable.

Thanks in advance!

Best regards

[Alberto_Umana]

Hello @yumnus,

Could you please share the full error trace? what is the package you are installing?

One suggestion, instead of relying on the metadata server, you can use a service account key file to authenticate with GCP Secret Manager. You can store the service account key as a Databricks secret and then use it in your code to authenticate. Here’s a general approach:

• Create a service account in GCP and download the JSON key file.
• Store the key file content as a secret in Databricks using the Databricks CLI or the Secrets API.
• Use the secret in your code to authenticate with GCP Secret Manager.

[yumnus]

Hi Alberto,

When we use No Isolation Shared Clusters it works, else:

the error messages:

WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 1 of 3. Reason: [Errno 111] Connection refused

WARNING:google.auth._default:No project ID could be determined. Consider running `gcloud config set project` or setting the GOOGLE_CLOUD_PROJECT environment variable

WARNING:google.auth.compute_engine._metadata:Compute Engine Metadata server unavailable on attempt 1 of 5. Reason: HTTPConnectionPool(host='metadata.google.internal', port=80): Max retries exceeded with url: /computeMetadata/v1/instance/service-accounts/default/?recursive=true (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7931121fe650>: Failed to establish a new connection: [Errno 111] Connection refused'))

google.auth.exceptions.TransportError: Failed to retrieve http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/?recursive=true from the Google Compute Engine metadata service. Compute Engine Metadata server unavailable

Thank you!

[yumnus]

Also the package google_cloud_secret_manager-2.22.0