GCE cluster chokes the secret api server.

Sadam97
New Contributor III

‎01-23-2025 04:08 AM

Hi,

We upgraded the GKE cluster to GCE cluster as per the databricks documentation. It works fine with one or two notebooks in job. Our production job has more than 40 notebooks and each notebook access the secret api and seems like secret api server gets choke with GCE cluster. Same job was/is running on GKE cluster with no issue.

0 Kudos

Alberto_Umana
Databricks Employee
Databricks Employee

‎01-23-2025 04:48 AM

Hi @Sadam97,

Could you please advise what is the error you are getting? 

Also the GKE version

0 Kudos

Sadam97
New Contributor III
In response to Alberto_Umana

‎01-23-2025 04:54 AM

Hi @Alberto_Umana ,

Error is on GCE and here is associated thread where i provided all info, https://community.databricks.com/t5/data-engineering/error-on-gce-job-cluster-after-upgrading-from-g...

0 Kudos

Alberto_Umana
Databricks Employee
Databricks Employee

‎03-21-2025 05:06 PM - edited ‎03-21-2025 05:08 PM

Hi @Sadam97,

This looks to be a known issue. I will share more details soon. There is a known issue with NAT and GKE and we’ll follow up offline

0 Kudos