topic Re: GCE cluster chokes the secret api server. in Data Engineering

GCE cluster chokes the secret api server.

Sadam97 — Thu, 23 Jan 2025 12:08:50 GMT

Hi,

We upgraded the GKE cluster to GCE cluster as per the databricks documentation. It works fine with one or two notebooks in job. Our production job has more than 40 notebooks and each notebook access the secret api and seems like secret api server gets choke with GCE cluster. Same job was/is running on GKE cluster with no issue.

Re: GCE cluster chokes the secret api server.

Alberto_Umana — Thu, 23 Jan 2025 12:48:56 GMT

Hi @Sadam97,

Could you please advise what is the error you are getting?

Also the GKE version

Re: GCE cluster chokes the secret api server.

Sadam97 — Thu, 23 Jan 2025 12:54:43 GMT

Hi @Alberto_Umana ,

Error is on GCE and here is associated thread where i provided all info, https://community.databricks.com/t5/data-engineering/error-on-gce-job-cluster-after-upgrading-from-gke/m-p/103056#M41318

Re: GCE cluster chokes the secret api server.

Alberto_Umana — Sat, 22 Mar 2025 00:08:44 GMT

Hi @Sadam97,

This looks to be a known issue. I will share more details soon. There is a known issue with NAT and GKE and we’ll follow up offline