cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Git Server Proxy won't stay up

Scanning
New Contributor II

We are running a job with multiple tasks that require the Git Server Proxy to remain operational for the entire duration of the job.

Since each task may need access to the proxy, and job runtimes vary from brief to several hours, what is the best approach to ensure the Git Server Proxy remains available throughout the job, aside from setting the auto-termination time to the maximum possible duration?

2 REPLIES 2

Stefan-Koch
Valued Contributor II

You could add a task, where you start/stop your proxy cluster with databricks cli.

StefanKoch_0-1739349722662.png

another consideration:
Do you even need the cluster if you're running a job? I assume that the job refers to code in a Git folder? Instead of the Git approach, you could deploy the code artifacts with asset bundles. That way, when you start a job, you don't need a Git proxy cluster

Scanning
New Contributor II

The Git Server Proxy is needed for various tasks within the job, and multiple jobs may rely on it. Manually terminating it at the end of job1 could lead to job2 failing.

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local communityโ€”sign up today to get started!

Sign Up Now