cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Restricting Workflow Creation and Implementing Approval Mechanism in Databricks

nistrate
New Contributor III

Hello Databricks Community,

I am seeking assistance understanding the possibility and procedure of implementing a workflow restriction mechanism in Databricks. Our aim is to promote a better workflow management and ensure the quality of the notebooks being attached to clusters.

Specifically, we want to restrict the ability for users to directly create workflows based on their user group. In our use case, if a user such as J.Doe is in the 'Data Engineer' group and is working on a new workflow, we would like them to submit the notebook for approval by an admin before it can be attached to a cluster.

Does Databricks support such a mechanism natively? If not, could you suggest any workarounds or third-party solutions that could help us achieve this? Ideally, we would like to automate this process as much as possible to avoid manual intervention.

Any guidance or suggestions would be greatly appreciated.

Thank you in advance for your time and help!

1 ACCEPTED SOLUTION

Accepted Solutions

User16502773013
Contributor II
Contributor II

Hello Nistrate,

If I understand the question correctly, the ask is to create an approval framework/workflow for workflows/jobs changes/commits, I don't believe this is currently supported however this can be supported through the use of source control like Git ( pull requests )

Databricks is also integrated with source control through Databricks repos (details here)

Also please find available jobs/workflow permissions for Databricks' admin here

Regards

View solution in original post

2 REPLIES 2

User16502773013
Contributor II
Contributor II

Hello Nistrate,

If I understand the question correctly, the ask is to create an approval framework/workflow for workflows/jobs changes/commits, I don't believe this is currently supported however this can be supported through the use of source control like Git ( pull requests )

Databricks is also integrated with source control through Databricks repos (details here)

Also please find available jobs/workflow permissions for Databricks' admin here

Regards

Avvar2022
Contributor

I believe this has to happen in 2 steps.

step1: Currently admin can't restrict workflow creation in databricks  currently any user with workspace access can create workflows. Admins should be able to restrict workflow creation. Databricks doesn't have this feature/functionality, this something simple to implement. not sure why they didn't.

Step2: First step eliminates all unwanted users. coming back to be able to review notebooks to be attached to clusters from workflow tasks. you can make use of git PR's and use databricks assets bundle to deploy code, workflows.

i think step 1 is must, unless admin able to restrict unwanted users from creating workflows any user can create workflow and the challenging part is user can create workflow in any workspace (prd,qas,dev,etc). Our workspaces are already messy with lot of workflows.  I hope databricks implements this feature soon.

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!