Dear all
we have a challenge. Developers create/recreate tables/views in PRD environment by running notebooks on all-purpose clusters where as the same notebooks already exist as jobs. Not sure, why the developers feel comfortable in using all-purpose clusters. The point now is that the objects get created/recreated with individual ids as owner and that is breaking our data flows as well as consumption flows at times, resulting into chaos. Had been created/recreated by a job, as the job runs as a service principal, and we want that service principal to be the owner of the objects in PRD env.
Any ideas on how i can overcome this obstacle? Can we use single user all-purpose clusters (with a service principal) be used by different individuals while invoking notebooks?
Appreciate any thoughts..