Hi @shweta_m 

I do agree with @juan_maedo ..

Just to add on top of it , answer of your solution is automated deployment pipelines embedded into your project repo using databricks asset bundles which is scalable and reliable way to implement permissions dynamically.

we dynamically create instance pools with right configuration defined under asset bundles files and grant 'can manage' permissions to this master SP (this is owned by our team as entire infra depends on this sp, basically owner of RG and part of service connection)

We also deploy databricks job using asset bundles (which use those instance pools to configure a job cluster) and make owner and runas to same master sp but we also add can manage permissions to this team ad group which is being synced from azure entraid to databricks.

As a side note apply tags to these instance pools which will help you to segregate cost per workload basis or domain basis , how you prefer.

Br