I wanted to ask this Q yesterday in the Q&A session with Mohan Mathews, but didn't get around to it (@Kaniz Fatmaโ do you know his handle here so I can tag him?)
We (and most development teams) have two environments: UAT/DEV and PROD. For those that don't know: UAT/DEV is for developing and testing software, and PROD is where code would be deployed and used by the customers/business.
I have the impression that this setup is considered industry standard, which most teams will accept without properly thinking "why". Is the added overhead of configuring a whole additional workspace and dealing with fiddly deployments worth it?
I once worked at a company that had very sensitive data, and this setup then made sense because the data on UAT would be mirrored from PROD but obfuscated, enabling more external contractors to be involved with the development of the application, without the added risk of data leaks.
But what if the data are not sensitive? Why not simply have a git branch called "prod", and "deploying" to that branch would just be a matter of merging "dev" branch into "prod"? I get the "what if the stack fails and business critical applications drop" argument... but:
1) how often will a Databricks workspace fail?
2) unless the applications being built and delivered to customers are truly critical to the business (and even a few minutes of down time would be very costly) - but I assume that for most companies that use Databricks this would not be the case.
3) does the idea of a workspace failing even make sense? In Databricks we have multiple compute clusters that can be defined. One could create a compute cluster called "PROD" only to be used for production applications...
Thank you very much in advance,
Oliver
