Hi all!
I'm appealing to all you folk who are clever than I for some advice on Databricks dev ops.
I was asked by my team leader to expand our singular environment to a devops style dev/pie/prd system, potentially using Dabs to promote code to higher environments.
I dived into tutorials, skilled up a bit and got to a reasonable understanding of Dabs along with a working example.
I then asked admin for two new workspaces to be created within our tenancy for pie and prd. However I was informed that this project was to be kept within our current single workspace and different catalogs or schemas or even tables created with _dev _pie _prod suffixes to represent the different environments.
To my mind this approach seems to take the wind out of the sails of using DABs since there are no target workspaces to deploy to.
I realise that this limited scenario can be achieved by any given notebook receiving a dev/pie/prd parameter from the job that is running it, thus giving it enough awareness of where to source and place data. This, I sense would require three identical jobs running the same notebook with the jobs named like so
myJob_dev, myJob_pie & myJob_prd
and each with an environment parameter set respectively.
This is in effect the same as those three jobs being deployed to target workspaces, but in this case they are simply sitting next to one another in the same workspace! Its a red flag to me though, because in the dabs world there would be a single yml definition of the job, and on deployment, target variables would set the parameters dynamically. In the single workspace environment, if say and extra task is added to the job it would have to be maintained in all three jobs and thus introducing the possibility of divergence and or course human error.
Is there still a way to use Dabs - or perhaps just a fruity git action to take a source definition of a job - one already running as a dev job - and "deploy" in other words "copy" that job create the required pie and prd jobs. Perhaps also with different schedules set and either in a paused or unpaused state depending on configuration.
Or.. am I overthinking this and there is a far simpler way to manage this entirely
TIA
Gary
