cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Git Server Proxy won't stay up

Scanning
New Contributor II

We are running a job with multiple tasks that require the Git Server Proxy to remain operational for the entire duration of the job.

Since each task may need access to the proxy, and job runtimes vary from brief to several hours, what is the best approach to ensure the Git Server Proxy remains available throughout the job, aside from setting the auto-termination time to the maximum possible duration?

2 REPLIES 2

Stefan-Koch
Valued Contributor II

You could add a task, where you start/stop your proxy cluster with databricks cli.

StefanKoch_0-1739349722662.png

another consideration:
Do you even need the cluster if you're running a job? I assume that the job refers to code in a Git folder? Instead of the Git approach, you could deploy the code artifacts with asset bundles. That way, when you start a job, you don't need a Git proxy cluster

Scanning
New Contributor II

The Git Server Proxy is needed for various tasks within the job, and multiple jobs may rely on it. Manually terminating it at the end of job1 could lead to job2 failing.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group