topic Re: Restricting Workflow Creation and Implementing Approval Mechanism in Databricks in Data Engineering

Restricting Workflow Creation and Implementing Approval Mechanism in Databricks

nistrate — Mon, 15 May 2023 18:40:32 GMT

Hello Databricks Community,

I am seeking assistance understanding the possibility and procedure of implementing a workflow restriction mechanism in Databricks. Our aim is to promote a better workflow management and ensure the quality of the notebooks being attached to clusters.

Specifically, we want to restrict the ability for users to directly create workflows based on their user group. In our use case, if a user such as J.Doe is in the 'Data Engineer' group and is working on a new workflow, we would like them to submit the notebook for approval by an admin before it can be attached to a cluster.

Does Databricks support such a mechanism natively? If not, could you suggest any workarounds or third-party solutions that could help us achieve this? Ideally, we would like to automate this process as much as possible to avoid manual intervention.

Any guidance or suggestions would be greatly appreciated.

Thank you in advance for your time and help!

Re: Restricting Workflow Creation and Implementing Approval Mechanism in Databricks

User16502773013 — Thu, 08 Jun 2023 02:13:42 GMT

Hello Nistrate,

If I understand the question correctly, the ask is to create an approval framework/workflow for workflows/jobs changes/commits, I don't believe this is currently supported however this can be supported through the use of source control like Git ( pull requests )

Databricks is also integrated with source control through Databricks repos (details here)

Also please find available jobs/workflow permissions for Databricks' admin here

Regards

Re: Restricting Workflow Creation and Implementing Approval Mechanism in Databricks

Avvar2022 — Fri, 07 Jun 2024 19:34:25 GMT

I believe this has to happen in 2 steps.

step1: Currently admin can't restrict workflow creation in databricks currently any user with workspace access can create workflows. Admins should be able to restrict workflow creation. Databricks doesn't have this feature/functionality, this something simple to implement. not sure why they didn't.

Step2: First step eliminates all unwanted users. coming back to be able to review notebooks to be attached to clusters from workflow tasks. you can make use of git PR's and use databricks assets bundle to deploy code, workflows.

i think step 1 is must, unless admin able to restrict unwanted users from creating workflows any user can create workflow and the challenging part is user can create workflow in any workspace (prd,qas,dev,etc). Our workspaces are already messy with lot of workflows. I hope databricks implements this feature soon.