topic Re: Customer deployment in Data Engineering

Customer deployment

thomasthomas — Fri, 14 Jan 2022 14:36:36 GMT

Hi,

I have a bunch of scripts in Databricks that perform a decent amount of data-wrangling. All of these scripts contain sensitive information and I have no intention of making them public.

I would like to provide a service to my customers - so they can use all these scripts - but they cannot see their content. My idea was to deploy Databricks in their own subscription but it looks like even with Databricks premium I cannot hide those scripts from my customers. The admin can see everything. (I wouldn't be the admin, obviously)

My second idea was to keep my databricks as is in my own sub and then create databricks clusters in their subscriptions. However, I can't see if there is such an option - if I happen to create a cluster, then it will run in my own subscription.

For obvious reasons, my clients do not want their data leave their subscription.

What's the recommended way of customer deployment, then? All of my scripts are pyspark scripts - so there is no obvious way to compile them given that python is an interpreted language. There are a few libraries out there but I don't trust them 100%.

Please advise,

Tamas

Re: Customer deployment

Anonymous — Mon, 24 Jan 2022 16:27:29 GMT

Hello, @Tamas D - My name is Piper, and I'm a community moderator for Databricks. Please accept my apologies for not responding sooner. 😞

I'll pass this to the subject matter experts now and the team will go looking for the best person to answer your question.

Re: Customer deployment

thomasthomas — Tue, 25 Jan 2022 10:04:08 GMT

@Piper Wilson Thank you.

In the meantime I found out that Databricks supports REST API calls to execute scripts in a given language but I am not convinced that that would be a great approach. With API 1.2 we need to define the language - and in almost all notebooks I keep switching between python and spark sql.

https://docs.databricks.com/dev-tools/api/1.2/index.html

Also, it not uncertain how temporary views would be handled if scripts would be sent separately. (Python, sql, python, sql etc.) Maybe it is not worth going down that road...

Re: Customer deployment

Anonymous — Tue, 25 Jan 2022 15:31:20 GMT

@Tamas D Thank you for the update.

Re: Customer deployment

Atanu — Wed, 26 Jan 2022 03:36:25 GMT

@Tamas D I understood your concern.

For cluster creation in different subscription I think that's by design at this moment. But I would like to request you to add your use case to https://feedback.azure.com/d365community/forum/2efba7dc-ef24-ec11-b6e6-000d3a4f0da0# which actually reviewed by our product team to evaluate and in future may be this can be implemented.
Also the case goes for hiding the script (if our ACLs does not solve the purpose)

Thanks. Please let me know if you have any queries.

Thanks