Databricks App with Proprietary packages and code

antoine_laurent
New Contributor II

Hello Databricks community,

I have built an application that has a lot of proprietary code and packages that I chose not to show for the public- I just want my service to be accessible for the people who install the application through a Streamlit frontend along with a few notebooks which are meant to be run as 'jobs' for setup and other processes. I'm planning to list my application on the marketplace, and I have a lot of the code and information ready. But I seem to be hitting a dead-end regarding what gets displayed in the user's workspace and catalogs. For example, my app has a bunch of UDFs, generates a few .whl files based on the compute architecture, and a few jobs which appear to be visible on my end when I test the application. My question would be to generally ask; how do I hide my code and packages from the installed user's range of visibility? As of now, I'm able to see most of my code and packages as I install them in my Databricks UI either through catalogs or jobs. Will Databricks make sure my code, and IP is secure while I launch my app to the marketplace? Databricks suggested the best practice to start an app would be to clone the code from a git repo. My repo is private, so how do I get a one-time setup so that only the application can clone my code into its compute and not show it to the user (I've considered GitHub pat tokens, but I just wanted some more ideas)? As a whole, I'm surely confused, at the end of the day, if a certain user accessing my application will be able to access any of the code via the jobs, packages via the volume (if i pip install during setup), or just any kind of IP as a whole? Additionally, I need help understanding the best practices for the following, it would be greatly appreciated: what are the best practices to deploy the code (other than GitHub) and can i just deploy my bundle, to install packages (like code artifact) or can I just plug it into my application somehow (I don't want to give my access keys/ expect my users to have AWS accounts to install my package), how can I make sure my code is not visible through a job and how can i access the user's compute outside the app's compute to run ML heavy tasks? At the end of the day, I want people to be able to use my service on their compute, but I don't want any of my code/data to be revealed to the user. I have a few questions in the ballpark of this area. Any help would genuinely be appreciated. 

Thanks, and Cheers,

Antoine