topic Re: git integration with volumes? in Data Engineering

git integration with volumes?

-werners- — Wed, 06 Sep 2023 10:21:43 GMT

Volumes in unity are said to be ideal for "storing library and config files of arbitrary formats such as .whl or .txt centrally and providing secure access across workspaces to it".

So basically this is the same we had with dbfs but with decent access control.
But if we want to use volumes for our libraries and config files, I am missing git integration.
How do we handle automated scenarios where a git provider automatically puts updated config files in the volumes on Unity?
In dbfs we could create a pipeline in f.e. Azure devops that runs when a merge is executed on main and copies all the concerning files to dbfs.
However for Volumes it is not clear if this is possible.
Anyone an idea/experience?

Re: git integration with volumes?

-werners- — Thu, 14 Sep 2023 07:54:44 GMT

I suspected as much. Otherwise the docs would mention it.
It would be great though to have some kind of automation, especially when working with config/libs.

Re: git integration with volumes?

ddundovic — Sat, 02 Aug 2025 17:40:10 GMT

This still doesn't seem to be possible. My current workflow for this is to add a task in my CI pipeline, where I use databricks-cli, i.e., `databricks fs cp /path/to/my/config/file dbfs:/Volume/path/to/databricks/volume` to copy my config files to the UC volume whenever they change.

Re: git integration with volumes?

Riverara — Mon, 04 Aug 2025 03:11:39 GMT

I totally get where you’re coming from! Managing config files with access control is crucial, but the lack of Git integration in Unity volumes can be a bit frustrating. In my experience, finding a smooth workflow can take time. I’ve found using tools like Solar Smash for quick simulations helps in environments where speed is key, but it's essential to ensure your file management system keeps up. Hopefully, there’s a solid solution on the horizon!