All Administration & Architecture posts

Re: Workload Identity Federation on ADO for account level provider

zubisid19 — Mon, 20 Jul 2026 19:34:51 GMT

You are using:

Azure-style host: https://accounts.azuredatabricks.net
Azure env vars: ARM_CLIENT_ID, ARM_TENANT_ID
Plus DATABRICKS_CLIENT_ID
That often makes the SDK try a different OIDC flow than intended (hitting /oidc/accounts/<id>/v1/token) and failing with EOF.

Also, your doc link is under .../aws/..., which may not match your Azure Databricks account setup.

Fix checklist
1) Use the correct cloud flow
If this is Azure Databricks, use Azure-specific auth guidance (not AWS path).

If this is actually Databricks on AWS, your account host should be:

https://accounts.cloud.databricks.com and you should not use ARM_* vars.
2) Do not mix auth methods
Pick one method only. For Azure workload identity, avoid mixing with unrelated Databricks OAuth vars.

Try removing DATABRICKS_CLIENT_ID first and keep only the Azure WIF-required vars + Databricks host/account settings.

3) Minimal account-level env (Azure pattern)
DATABRICKS_HOST=https://accounts.azuredatabricks.net
DATABRICKS_ACCOUNT_ID=<account_id>
Azure WIF vars from your pipeline/service connection (client/tenant/OIDC token)
Optional: set explicit auth type if your tooling supports it
4) Validate with CLI debug
Run below cli command -
databricks auth env --debug
Then test an account API call:

cli command -
databricks account users list --debug

5) If still EOF, check network/proxy
EOF can also be proxy/TLS interruption to accounts.azuredatabricks.net. Test from agent:

cli command -
curl -v https://accounts.azuredatabricks.net

If blocked, fix proxy/no_proxy/firewall rules.

Workload Identity Federation on ADO for account level provider

eshwari — Mon, 20 Jul 2026 17:09:21 GMT

I am using Workload Identity Federation for ADO pipelines to authenticate to databricks.

I followed https://docs.databricks.com/aws/en/dev-tools/auth/provider-azure-devops

It works with workspace url but doesn't work with account level url.

I get error: failed during request visitor: default auth: Post "https://accounts.azuredatabricks.net/oidc/accounts/<accountid>/v1/token": Post "https://accounts.azuredatabricks.net/oidc/accounts/<accountid>/v1/token": EOF. Config: host=https://accounts.azuredatabricks.net, account_id=<accountid>, azure_client_id=***, azure_tenant_id=<tenantid>, client_id=***. Env: ARM_CLIENT_ID, ARM_TENANT_ID, DATABRICKS_CLIENT_ID

Re: Databricks App with Proprietary packages and code

GabFernandes — Thu, 16 Jul 2026 16:39:37 GMT

Hey Antoine, great questions — this is a common challenge when building commercial apps on Databricks. Let me address each area directly.

The fundamental tradeoff

The core tension here is architectural: when code runs in the user's workspace, it's very hard to fully hide it. Databricks runs the code on their compute, which means the code has to be there in some form. The cleanest way to protect your IP is to keep the sensitive logic in your infrastructure and expose only an API surface to the user's workspace.

For the Streamlit app (Databricks Apps)

The app source files (app.py, dependencies) are deployed into the user's workspace and are visible. To minimize exposure: compile your sensitive logic into a pre-built .whl using Cython (compiles Python to C extensions — much harder to reverse than bytecode) and reference it as a dependency instead of shipping raw .py files. For private package distribution, set PIP_INDEX_URL in your app.yaml pointing to a private registry (Azure Artifacts, GitHub Packages, Nexus/Artifactory) and store the credentials as a Databricks Secret referenced via valueFrom — no hardcoded tokens, no AWS keys required.

For jobs and notebooks

Any notebook or file attached to a job in the user's workspace is visible to that user. The mitigation is the same: replace raw Python source with a pre-built wheel installed at runtime. The job then imports from the wheel rather than running readable source files.

For UDFs

UDF bodies registered in Unity Catalog are stored and queryable — users with USE SCHEMA can inspect the definition. If your UDF logic is the IP, consider wrapping it in a model serving endpoint call instead of implementing it directly as a SQL/Python UDF.

For ML-heavy tasks

This is actually where Databricks gives you the cleanest story: host your ML models as Model Serving endpoints in your own workspace and expose them via REST. The user's app calls https://your-workspace.azuredatabricks.net/serving-endpoints/your-model/invocations — the model runs on your compute, the user never sees the code or weights, and you control access via API tokens.

For the git/credentials problem

Don't clone at install time if you can avoid it. Pre-build your artifacts (wheels, config files) and distribute them through your private registry. If you absolutely need a git clone step, use Databricks Secrets to store the PAT and reference it at runtime — but know that a sufficiently privileged user on the cluster can read secrets, so this only reduces exposure, it doesn't eliminate it.

Bottom line on Marketplace IP

Databricks does conduct a security review before publishing, but they don't encrypt or obfuscate deployed code on your behalf. The protection is your responsibility at the architecture level. The pattern that genuinely protects IP is: pre-compiled artifacts for the client-side code + remote API/serving endpoints for anything compute-sensitive.

Hope this helps clarify the landscape — happy to go deeper on any of these.

Databricks App with Proprietary packages and code

antoine_laurent — Thu, 16 Jul 2026 16:32:49 GMT

Hello Databricks community,

I have built an application that has a lot of proprietary code and packages that I chose not to show for the public- I just want my service to be accessible for the people who install the application through a Streamlit frontend along with a few notebooks which are meant to be run as 'jobs' for setup and other processes. I'm planning to list my application on the marketplace, and I have a lot of the code and information ready. But I seem to be hitting a dead-end regarding what gets displayed in the user's workspace and catalogs. For example, my app has a bunch of UDFs, generates a few .whl files based on the compute architecture, and a few jobs which appear to be visible on my end when I test the application. My question would be to generally ask; how do I hide my code and packages from the installed user's range of visibility? As of now, I'm able to see most of my code and packages as I install them in my Databricks UI either through catalogs or jobs. Will Databricks make sure my code, and IP is secure while I launch my app to the marketplace? Databricks suggested the best practice to start an app would be to clone the code from a git repo. My repo is private, so how do I get a one-time setup so that only the application can clone my code into its compute and not show it to the user (I've considered GitHub pat tokens, but I just wanted some more ideas)? As a whole, I'm surely confused, at the end of the day, if a certain user accessing my application will be able to access any of the code via the jobs, packages via the volume (if i pip install during setup), or just any kind of IP as a whole? Additionally, I need help understanding the best practices for the following, it would be greatly appreciated: what are the best practices to deploy the code (other than GitHub) and can i just deploy my bundle, to install packages (like code artifact) or can I just plug it into my application somehow (I don't want to give my access keys/ expect my users to have AWS accounts to install my package), how can I make sure my code is not visible through a job and how can i access the user's compute outside the app's compute to run ML heavy tasks? At the end of the day, I want people to be able to use my service on their compute, but I don't want any of my code/data to be revealed to the user. I have a few questions in the ballpark of this area. Any help would genuinely be appreciated.

Thanks, and Cheers,

Antoine

Re: No Egress for Serverless Workspace

Louis_Frolio — Wed, 15 Jul 2026 19:59:47 GMT

Greetings @Janis404 ,

Nice work pulling system.access.outbound_network and narrowing this to serverless. That table is where the answer lives, so let me walk through what it's telling you.

Read the log rows first:

access_type is DROP, which is real enforcement, not a dry-run entry (dry-run shows up as DRY_RUN_DENIAL). So egress is being blocked right now, not merely recorded.
The rcode: 5 in the dns_event is a DNS REFUSED. The serverless DNS layer refused the lookup, and that is exactly what produces the Python gaierror [Errno -2] Name or service not known. Your code is fine. The block sits upstream of Python.
One nuance: denials are logged to this table even when a policy is in Full access mode, so the presence of rows isn't the smoking gun by itself. The DROP verdict is.

Now the apparent contradiction: the policy screenshot says "Allow access to all destinations," but google.com, a plain public name, is being dropped. Under a Full access policy that's actually in force, that lookup should succeed. Two things explain the gap, and I'd rule them out in this order.

The config you're showing may not be the config that was active when those DROP rows were written. Your policy view and your log output aren't necessarily from the same moment, so pin them together: terminate the serverless environment on the notebook (or start a brand-new serverless run rather than reusing one that's still warm), re-run the socket test immediately, then query outbound_network again and confirm whether there are fresh DROP rows for google.com with new timestamps. This also handles the warm-compute angle: network access mode changes apply to newly provisioned serverless compute, not to a session that was already running when you flipped the setting. The "wait 15 minutes" advice was reaching for this, but 15 minutes is optimistic, and a session that never fully recycles can lag much longer.
If a fresh session still drops it, have an account admin inspect the policy through the account API, CLI, or Terraform and confirm restriction_mode is FULL_ACCESS with no blocked_internet_destinations entries. Databricks lets you block specific domains even while a policy is in Full access mode, so the General tab's radio button doesn't tell the whole story. Check for an entry matching google.com or a broader parent domain. One caveat: a single blocklist entry wouldn't explain your own domains getting dropped too, so if everything is failing, the timing issue above is the likelier culprit than a targeted block.

On the NCC: since you called it out, salfy-prod-ncc is almost certainly not behind these particular rows. A private endpoint rule for your Postgres routes that one FQDN privately by matching its hostname; it doesn't produce a DROP for an unrelated public name like google.com in the egress table. The DROP is an egress policy decision, so that's where to look, not the private-connectivity path.

If you've confirmed Full access on genuinely fresh serverless compute, with no blocked destinations, and it still drops google.com, that no longer matches expected Full access behavior and it's a Support case. Give them everything on the first pass: workspace ID (dbc-6b9d00ae-0ac6), region eu-central-1, the policy name, the outbound_network rows with their timestamps, and a failing run ID, and state plainly that you're serverless-only and Full access is set.

Takeaway: the Python gaierror is a symptom, not the problem. Your logs already show the serverless egress layer refusing DNS, so the fix is getting a Full access policy actually in force on fresh compute, confirmed by fresh DROP-free rows, before you change anything else. Start with a cold session and matching timestamps.

Regards,
Louis

Re: X-Forwarded-Access-Token occasionally return stale values from a previous session

iyashk-DB — Wed, 15 Jul 2026 17:54:36 GMT

This almost always turns out to be st.cache_resource on the connection, not the reverse proxy reusing anything.

You said you checked your st.cache_data functions and they key on username, which is the right instinct, but the leak usually hides in whatever function opens the SQL connection or session object. st.cache_resource caches are globally by default, they're shared across every user and every session in the app process, not per-session like st.session_state. The cache key comes from the function's arguments, so if your connection-builder function looks like this:

@st.cache_resource(ttl=300)
def get_connection():
    return sql.connect(server_hostname=..., http_path=..., access_token=user_access_token)

every user hits the same cached connection object for up to the TTL window, and that connection was opened with whichever token happened to trigger the first cache miss. That's exactly the symptom you're describing: queries occasionally running under a previous user's identity, and it'll look "random" because it only shows up when the cache is warm from someone else's request.

The fix is to make the token part of the cache key so each identity gets its own cached object:

@st.cache_resource(ttl=300)
def get_connection(user_token: str):
    return sql.connect(server_hostname=..., http_path=..., access_token=user_token)

conn = get_connection(user_access_token)

Or simplest and safest, don't cache the connection at all, open it fresh per request and close it after the query, which is what Databricks' own Apps examples do for this exact reason.

To your second question, yes, there's a cheap way to verify the token belongs to who you think before you use it. Use the SDK with that specific token and check the identity it resolves to against X-Forwarded-Email:

from databricks.sdk import WorkspaceClient

w = WorkspaceClient(host=host, token=user_access_token)
me = w.current_user.me()
if me.user_name != username:
    st.error("Session identity mismatch, please refresh.")
    st.stop()

One more thing worth knowing separately from the caching issue: st.context.headers itself is only populated once, at the initial page load / WebSocket connection, and doesn't refresh on reruns. So on a long-lived tab, even without any caching bug, the token you're holding can go stale if the underlying session persists across a reconnect. The identity check above catches that case too, since it fails closed instead of quietly running a query under a token that no longer matches.

X-Forwarded-Access-Token occasionally return stale values from a previous session

tesshy-amane — Wed, 15 Jul 2026 10:52:39 GMT

Hi all,

I'm running a Streamlit app on Databricks Apps with On-Behalf-Of User Authorization enabled. We've observed that X-Forwarded-Access-Token and X-Forwarded-Email sometimes return values that belong to a previous session(logged out other user, using same browser window) rather than the current one, e.g.:

python

user_access_token = st.context.headers.get("X-Forwarded-Access-Token")
username = st.context.headers.get("X-Forwarded-Email")

username occasionally appears to be a different user's email from an earlier session, and the associated user_access_token then gets used for a Databricks SQL query executed on behalf of the wrong identity.

What we've verified:

We are not caching the connection or the token anywhere in our code (no st.cache_resource wrapping a connection object, no module-level/global variables holding the token). user_access_token and username are read fresh from st.context.headers at the top of the script on every run.
Our st.cache_data-decorated functions include username as an explicit cache key argument, so we don't believe this is Streamlit's function-level caching bleeding across users.

Question:

Is it expected/known that the Databricks Apps reverse proxy (or the underlying connection between the proxy and the app container) can reuse a connection across different end-user sessions in a way that causes X-Forwarded-* headers from one user's request to be forwarded on a subsequent request from a different user?
Is there a recommended way to positively verify, on every request, that X-Forwarded-Access-Token actually corresponds to X-Forwarded-Email (e.g., calling current_user.me() with the token) before we trust it for a query? We've added this as a client-side safety check in the meantime, but would prefer a platform-level explanation/fix if this is a known proxy behavior.

Any insight into whether this is expected behavior, a known issue, or something we should file a support ticket for would be appreciated. Happy to share more app code / logs if useful.

Thanks!

Re: How to change passkey device for partner-academy.databricks.com?

Fikrat — Tue, 14 Jul 2026 15:12:07 GMT

Thanks, @Advika , I'll follow thaT advice.

Re: How to change passkey device for partner-academy.databricks.com?

Advika — Tue, 14 Jul 2026 14:52:46 GMT

Hello @Fikrat,

Please raise a ticket with the Databricks Support team. They'll be able to help you remove or reset your passkey.

Re: enable databricks in azure for the whole tenant

satycse06 — Tue, 14 Jul 2026 07:42:18 GMT

@szymon_dybczak Thanks for the response and well explanation.

I have another question related to vnet injection architecture. So as per you we can create a management group and include all the subscription in that management group and will apply the polices at root level. But what about network shall I assign each vnet for each of the subscription which will be part of the mangement group or a hub spoke architecture so that we can manage the network policies centrally like not allowing the public IP accessibility of databricks workspace

Re: Oracle Connectivity - CDC

harisrinivasay — Tue, 14 Jul 2026 06:19:47 GMT

Answer: Not yet.

As of now, Oracle is not listed as a supported CDC (Change Data Capture) database connector in Lakeflow Connect managed CDC connectors.

Databricks has recently introduced Query-Based Connectors for databases including Oracle. These connectors are different from CDC:

They query the source database on a schedule.
They use a cursor/watermark column (for example, LAST_UPDATED_DATE).
They do not read Oracle redo logs or perform transaction-log-based CDC.
They do not require an ingestion gateway or CDC configuration on the source database.

Re: Oracle Connectivity - CDC

psingh13 — Tue, 14 Jul 2026 04:05:49 GMT

Hi,

For SQL server it is clear, I was asking about Oracle.

Thanks.

Re: Genie Code Usage Dashboard

rkhbo3003 — Tue, 14 Jul 2026 03:41:39 GMT

I built the same however i paramterized queries based on workspace and even implemented RLS to that , we have even implemented alerts etc

Re: Oracle Connectivity - CDC

rkhbo3003 — Tue, 14 Jul 2026 03:39:45 GMT

Hi ,

Are you using lakeflow ? For sql there is lakeflow connector it uses . I have used that it us very convenient however need to do configuration at account level

Re: Using Terraform to GRANT SELECT ON ANY FILE securable

TMD — Mon, 13 Jul 2026 18:54:52 GMT

thanks. this works even though TF takes unusually very long time to apply the changes - 12m

How to change passkey device for partner-academy.databricks.com?

Fikrat — Mon, 13 Jul 2026 18:34:07 GMT

Hi there,

I've registered to partner-academy.databricks.com with my Slalom account, and it requires mobile phone passkey on each login. The problem is the passkey I've configured long ago is on my old phone and I can't find place in that web site that allows changing that. Any help would be much appreciated!

Re: Clarification on Auto Loader Managed File Events with Unity Catalog Managed Volumes

ShamenParis — Mon, 13 Jul 2026 15:04:51 GMT

The confusion here usually stems from overlapping two different architectural concepts: the "Landing Zone" (where raw data arrives) and "Managed Storage" (where Databricks governs the data lifecycle).

To address your questions directly:

1. Does Databricks configure S3 Event Notifications for the entire prefix, including __unitystorage? Yes. When you enable File Events on an External Location at the cloud level, AWS S3 simply configures EventBridge/SQS for that entire prefix. Because __unitystorage sits under that prefix, S3 will technically generate cloud events for files added to Managed Volumes/Tables within that path.

2. Can Auto Loader use Managed File Events to ingest from a Managed Volume? While technically events might be firing in the cloud, using Managed File Events to ingest from a Managed Volume is an anti-pattern and is not the intended use case. Managed File Events are officially designed to ingest data from External Locations/External Volumes. By definition, Databricks governs the lifecycle of Managed Volumes. External systems (like your AWS Kinesis firehose, SFTP drops, or third-party apps) should never be dropping raw files directly into the obscure __unitystorage path. Therefore, Auto Loader shouldn't need to listen for external file arrival events in a Managed Volume.

3. What is the purpose of enabling File Events on an External Location used for Managed Storage? Ideally, you shouldn't be doing this. It is a best practice to strictly separate your storage buckets (or at least your top-level prefixes):

Landing/Raw Bucket: Configured as an External Location with File Events enabled. External systems drop data here.
Managed Bucket: Configured as the root storage for your Unity Catalog/Schemas. File Events do not need to be enabled here because Databricks orchestrates all data movement into this location.

4. Is your recommended architecture correct? Yes, your assumption at the end is 100% correct. The best-practice design pattern is:

Ingestion: External Volume (pointing to your landing bucket) + Managed File Events -> Auto Loader -> Target Delta Table.
Internal Storage: Managed Volumes/Tables are used for internal Databricks processing, intermediate steps, or saving ML artifacts. If you ever do need to read from a Managed Volume with Auto Loader, Directory Listing is the appropriate method, as the volume of files should be entirely controlled by internal Databricks processes.

Re: Clarification on Auto Loader Managed File Events with Unity Catalog Managed Volumes

Ashwin_DSA — Mon, 13 Jul 2026 13:50:58 GMT

Hi @APJESK,

I am wondering if the main source of confusion is the name cloudFiles.useManagedFileEvents. In the public docs, "managed" here refers to the Databricks-managed file-events service and cache layer, not to Unity Catalog managed volumes or the __unitystorage path. The Auto Loader with file events overview explains that Databricks sets up cloud resources to listen for file events, processes those notifications, and then Auto Loader reads from that file-events cache when cloudFiles.useManagedFileEvents=true is set.

On your first question... at a high level, file events are configured at the external-location scope. The file notification mode docs say Databricks uses a single file notification queue for all streams from a given external location, and the manage external locations docs say enabling file events on an external location allows Databricks to configure S3-side notifications for that location. So, if your managed storage is physically landing under s3://my-bucket/catalog/cat1/__unitystorage/..., as in your example, then it is still under the broader external-location prefix s3://my-bucket/catalog/cat1/ that you described. Because of that, object creates under __unitystorage would be within that same underlying external-location notification scope. That said, the public docs do not present __unitystorage as the intended Auto Loader ingestion target.

On your second question...I would be careful not to treat that as official support for managed volumes as a managed-file-events source. The public docs consistently describe the source as an external location or a volume on an external location, and they specifically recommend creating a separate external volume for each ingest path and passing the volume path to Auto Loader. I do not see public documentation that explicitly positions managed volumes as an officially supported Auto Loader source for cloudFiles.useManagedFileEvents=true. If you do see that, can you please post the link to that page?

There is also a related signal in the file arrival trigger docs... they explicitly say the monitored path must not contain managed locations of catalogs and schemas. That is a different feature from Auto Loader itself, but it points in the same direction and reinforces that Databricks is careful about treating Unity Catalog managed locations as event-driven file-ingestion sources.

On your third question...the answer is that enabling file events is still valuable for workloads that read paths governed by that external location. The docs say it makes Auto Loader setup easier, enables incremental file discovery with notification-like performance, and improves file-arrival triggers for jobs. In other words, enabling file events on the external location is about improving ingestion and event-driven processing for that external-location-backed storage boundary. It is not the same thing as saying every path underneath that boundary is a recommended Auto Loader source.

Hope this clarifies your questions.

If this answer resolves your question, could you mark it as “Accept as Solution”? That helps other users quickly find the correct fix.

Re: enable databricks in azure for the whole tenant

szymon_dybczak — Mon, 13 Jul 2026 13:43:48 GMT

Hi @satycse06 ,

Azure Databricks should be onboarded through a combination of Azure tenant/subscription controls, the Azure Databricks account console, and standardized workspace deployments. There is not one tenant-wide “Enable Databricks” switch that automatically governs every workspace.

1. Initial tenant and account setup

Select the Azure subscriptions that are permitted to host Databricks.
Register the Microsoft.Databricks resource provider only in the approved subscriptions. Registration is performed per subscription, not once for the entire tenant. Microsoft recommends registering resource providers only when they are ready to be used.
Create the first controlled Databricks workspace.
The deployment identity needs Azure Contributor/Owner at subscription level,
Establish the Databricks account administrators.
For the initial account bootstrap, a Microsoft Entra Global Administrator signs in to the Databricks account console and becomes the first Databricks account administrator. That person should then delegate the Account Admin role to two or three dedicated platform administrators and remove the Global Administrator from routine Databricks administration.
Use Microsoft Entra ID for authentication and Conditional Access.
Microsoft Entra ID single sign-on is available by default for both the Databricks account console and workspaces. MFA and Conditional Access should therefore be enforced through Entra ID.

About the Enterprise Application

The AzureDatabricks enterprise application, whose standard resource/application ID is 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d, is the Microsoft first-party Databricks service identity. It should not be treated as the main mechanism for assigning access to individual workspaces. Workspace access is managed through Databricks account-level identities, groups and workspace assignments.

For a new account, use Automatic Identity Management, which is enabled by default for accounts created after August 1, 2025. It uses Entra ID as the source of record and supports users, service principals, groups and nested groups. If Automatic Identity Management is unavailable for a legacy account, configure the separate Azure Databricks SCIM Provisioning Connector enterprise application at account level.

Users should be assigned to workspaces through these account-level groups rather than individually.

2. Enabling Databricks only in selected subscriptions

Resource-provider registration alone is not sufficient as a security boundary because Azure Contributor and Owner roles include the permission to register resource providers.

The recommended control is:

Place approved Databricks subscriptions under a dedicated Azure management group.
Apply an Azure Policy at the parent or root management-group level that denies Microsoft.Databricks/workspaces.
Exclude only the approved Databricks management group or subscriptions from that deny assignment.
Grant workspace deployment permissions only to the central platform team or deployment service principal in those subscriptions.
Deploy workspaces exclusively through an approved Terraform, Bicep or ARM module.

If my answer was helpful, please consider marking it as accepted solution.

Re: SPOG discovery probe failed

Miss_Maee — Mon, 13 Jul 2026 12:28:41 GMT

In simple terms, this error means:

Your computer tried to connect to your Databricks workspace, but it couldn't verify the website's security certificate.
The important part is:
certificate verify failed:
self-signed certificate in certificate chain
 This usually happens because:
You're connected to a company VPN or corporate network that inspects HTTPS traffic.
A proxy or antivirus is replacing the website's security certificate.
Your computer is missing the required trusted certificates.
The "SPOG discovery" message isn't the actual problem—it's just telling you that Databricks couldn't check some workspace information because the secure connection failed.
What you can do
Open your Databricks workspace URL in a browser:
https://adb-1235610976271489.3.azuredatabricks.net
If it opens normally, your workspace is likely fine.
If you see a certificate warning, it's a network or certificate issue.
If you're using a company VPN or proxy, try disconnecting and connecting again, or ask your IT team if they use SSL/HTTPS inspection.
Make sure your Databricks tools (CLI, SDK, or VS Code extension) are up to date.