GCE cluster chokes the secret api server.

[Sadam97]

Hi,

We upgraded the GKE cluster to GCE cluster as per the databricks documentation. It works fine with one or two notebooks in job. Our production job has more than 40 notebooks and each notebook access the secret api and seems like secret api server gets choke with GCE cluster. Same job was/is running on GKE cluster with no issue.

[Alberto_Umana]

Hi @Sadam97,

Could you please advise what is the error you are getting? 

Also the GKE version

[Sadam97]

Hi @Alberto_Umana ,

Error is on GCE and here is associated thread where i provided all info, https://community.databricks.com/t5/data-engineering/error-on-gce-job-cluster-after-upgrading-from-g...

[Alberto_Umana]

Hi @Sadam97,

This looks to be a known issue. I will share more details soon. There is a known issue with NAT and GKE and we’ll follow up offline