topic Re: How to get git commit ID of the repository the script runs on? in Data Engineering

How to get git commit ID of the repository the script runs on?

annagriv — Thu, 11 May 2023 19:30:02 GMT

I have a script in a repository on DataBricks. The script should log the current git commit ID of the repository. How can that be implemented?

I tried various command, for example:

result = subprocess.run(['git', 'rev-parse', 'HEAD'], stdout=subprocess.PIPE, check=True)

but getting the following error:

Command '['git', 'rev-parse', 'HEAD']' returned non-zero exit status 128.

Any help with this will be appreciated 🙏

Re: How to get git commit ID of the repository the script runs on?

-werners- — Fri, 12 May 2023 08:18:42 GMT

I think this is because of the fact that the code and execution (clusters) are separated.

Re: How to get git commit ID of the repository the script runs on?

annagriv — Sun, 14 May 2023 15:20:47 GMT

Thanks for you reply, so this means its impossible to log the git commit id? There is no way to pass this information to the cluster?

Re: How to get git commit ID of the repository the script runs on?

-werners- — Mon, 15 May 2023 07:13:05 GMT

I think that is indeed the case (not 100% sure as I do not know the innards of Databricks).

The on-demand cluster receives a program to be executed and I don't think a git history is passed.

Re: How to get git commit ID of the repository the script runs on?

rich_avery — Thu, 20 Feb 2025 15:09:13 GMT

As of February 2025 it is possible using the repos rest api (I use the sdk for simplicity) and dbutils. This code snippet assumes that the notebook it's being run in is in the root of the databricks repos folder that we're interested in.

from databricks.sdk import WorkspaceClient

w = WorkspaceClient()

notebook_path = dbutils.notebook.entry_point.getDbutils().notebook().getContext().notebookPath().get()

prefix_path = '/'.join(notebook_path.split('/')[:-1])

repo_info = list(w.repos.list(path_prefix = prefix_path))[0]

repo_info.head_commit_id

Databricks SDK for Python (Beta) — Databricks SDK for Python beta documentation

w.repos: Repos — Databricks SDK for Python beta documentation

Re: How to get git commit ID of the repository the script runs on?

vr — Fri, 07 Mar 2025 02:38:02 GMT

This should be the accepted answer.

A bit shorter version:

import os from databricks.sdk import WorkspaceClient w = WorkspaceClient() repo_info = next(w.repos.list(path_prefix=os.getcwd())) print(repo_info.head_commit_id)

Re: How to get git commit ID of the repository the script runs on?

bestekov — Wed, 16 Apr 2025 14:54:33 GMT

Here is a version of @vr 's solution that can be run from any folder within the rep. It uses regex to extract the root from the path in the form of \Repos\<username>\<some-repo:

import os import re from databricks.sdk import WorkspaceClient w = WorkspaceClient() repo_root = re.search(r'\/Repos\/[^\/]+\/[^\/]+', os.getcwd()).group(0) repo_info = next(w.repos.list(path_prefix=repo_root)) print(repo_info.head_commit_id)