Databricks App with Proprietary packages and code

[antoine_laurent]

Hello Databricks community,

I have built an application that has a lot of proprietary code and packages that I chose not to show for the public- I just want my service to be accessible for the people who install the application through a Streamlit frontend along with a few notebooks which are meant to be run as 'jobs' for setup and other processes. I'm planning to list my application on the marketplace, and I have a lot of the code and information ready. But I seem to be hitting a dead-end regarding what gets displayed in the user's workspace and catalogs. For example, my app has a bunch of UDFs, generates a few .whl files based on the compute architecture, and a few jobs which appear to be visible on my end when I test the application. My question would be to generally ask; how do I hide my code and packages from the installed user's range of visibility? As of now, I'm able to see most of my code and packages as I install them in my Databricks UI either through catalogs or jobs. Will Databricks make sure my code, and IP is secure while I launch my app to the marketplace? Databricks suggested the best practice to start an app would be to clone the code from a git repo. My repo is private, so how do I get a one-time setup so that only the application can clone my code into its compute and not show it to the user (I've considered GitHub pat tokens, but I just wanted some more ideas)? As a whole, I'm surely confused, at the end of the day, if a certain user accessing my application will be able to access any of the code via the jobs, packages via the volume (if i pip install during setup), or just any kind of IP as a whole? Additionally, I need help understanding the best practices for the following, it would be greatly appreciated: what are the best practices to deploy the code (other than GitHub) and can i just deploy my bundle, to install packages (like code artifact) or can I just plug it into my application somehow (I don't want to give my access keys/ expect my users to have AWS accounts to install my package), how can I make sure my code is not visible through a job and how can i access the user's compute outside the app's compute to run ML heavy tasks? At the end of the day, I want people to be able to use my service on their compute, but I don't want any of my code/data to be revealed to the user. I have a few questions in the ballpark of this area. Any help would genuinely be appreciated. 

Thanks, and Cheers,

Antoine

[GabFernandes]

Hey Antoine, great questions — this is a common challenge when building commercial apps on Databricks. Let me address each area directly.

The fundamental tradeoff

The core tension here is architectural: when code runs in the user's workspace, it's very hard to fully hide it. Databricks runs the code on their compute, which means the code has to be there in some form. The cleanest way to protect your IP is to keep the sensitive logic in your infrastructure and expose only an API surface to the user's workspace.

For the Streamlit app (Databricks Apps)

The app source files (app.py, dependencies) are deployed into the user's workspace and are visible. To minimize exposure: compile your sensitive logic into a pre-built .whl using Cython (compiles Python to C extensions — much harder to reverse than bytecode) and reference it as a dependency instead of shipping raw .py files. For private package distribution, set PIP_INDEX_URL in your app.yaml pointing to a private registry (Azure Artifacts, GitHub Packages, Nexus/Artifactory) and store the credentials as a Databricks Secret referenced via valueFrom — no hardcoded tokens, no AWS keys required.

For jobs and notebooks

Any notebook or file attached to a job in the user's workspace is visible to that user. The mitigation is the same: replace raw Python source with a pre-built wheel installed at runtime. The job then imports from the wheel rather than running readable source files.

For UDFs

UDF bodies registered in Unity Catalog are stored and queryable — users with USE SCHEMA can inspect the definition. If your UDF logic is the IP, consider wrapping it in a model serving endpoint call instead of implementing it directly as a SQL/Python UDF.

For ML-heavy tasks

This is actually where Databricks gives you the cleanest story: host your ML models as Model Serving endpoints in your own workspace and expose them via REST. The user's app calls https://your-workspace.azuredatabricks.net/serving-endpoints/your-model/invocations — the model runs on your compute, the user never sees the code or weights, and you control access via API tokens.

For the git/credentials problem

Don't clone at install time if you can avoid it. Pre-build your artifacts (wheels, config files) and distribute them through your private registry. If you absolutely need a git clone step, use Databricks Secrets to store the PAT and reference it at runtime — but know that a sufficiently privileged user on the cluster can read secrets, so this only reduces exposure, it doesn't eliminate it.

Bottom line on Marketplace IP

Databricks does conduct a security review before publishing, but they don't encrypt or obfuscate deployed code on your behalf. The protection is your responsibility at the architecture level. The pattern that genuinely protects IP is: pre-compiled artifacts for the client-side code + remote API/serving endpoints for anything compute-sensitive.

Hope this helps clarify the landscape — happy to go deeper on any of these.