Pipeline still needs USE SCHEMA on an old schema it no longer writes to

[keshavmonga22]

We have a Lakeflow (DLT) pipeline that runs as a task inside a job. A service principal is set as the run-as identity.

The pipeline originally created its streaming tables and a metadata Delta table (which gets updated in a separate task in the job) in one schema. That was changed a long time ago and everything now points to a new schema. There is no reference to the old schema anywhere in the source code.

Even so, updates keep failing because a new service principal was created which doesn't have USE SCHEMA on the old schema.

What I've already checked:

• The tables sitting in the old schema had pipelines.pipelineId set to this pipeline, so DLT still treated them as its own.
• The pipeline's lineage view still showed 2 tables from the old schema.
• I dropped those tables manually. The error still shows up.
• I ran a full refresh. Same error.
• I can't query the event log directly since the TVF is owner-restricted, so I can't see which object it's actually failing on.

Is there anything else that keeps a reference to the old schema once the tables themselves are gone? Event log destination, Auto Loader schema/checkpoint locations, internal materialization schemas, something else? Or is permanently granting USE SCHEMA on the old schema the only realistic option here?

Unity Catalog is enabled, pipeline is triggered from a job task, run-as is a service principal.

#Lakeflow #DLT #DeltaLiveTables #UnityCatalog

[adnan_alvee]

I don’t think the old schema grant is necessarily the only long-term answer.

Even after the original streaming tables are dropped, a Lakeflow pipeline can still have hidden artifacts that reference the prior schema. In practice, the main things I’d look at are the pipeline event log location, any hidden backing/materialization objects, and if Auto Loader is involved, the schema metadata that follows the event log location.

For Unity Catalog pipelines, changing the default catalog or schema causes the previous tables to become inactive, and dropping those old tables is the right cleanup step. But that cleanup does not by itself rule out other pipeline-managed metadata still tied to the old schema.

One detail that stands out is the event log behavior. By default, the pipeline writes its event log to a hidden Delta table in the configured catalog/schema, and Databricks specifically notes that removing the event log or its parent catalog/schema can cause future updates to fail. Databricks also documents that the event log location serves as the schema location for Auto Loader queries in the pipeline.

Depending on the pipeline’s publishing mode, there may also be hidden backing tables involved. In the current/default architecture, these hidden backing tables and event-log objects live alongside the user-facing schema objects. In legacy publishing mode, they instead live under internal pipeline-managed schemas in __databricks_internal.

So my read is that there are still a few realistic places where the old schema could be referenced, even though it no longer appears in the source code or in the active published tables.

As a practical next step, I’d suggest checking the pipeline settings/API response for the event log configuration and publishing mode, and then looking for any hidden event-log or materialization objects that still exist in the old schema.

A few references that may be helpful:

• Use Unity Catalog with pipelines

• Pipeline event log

• Configure pipelines

• Enable the default publishing mode in a pipeline

[binlogreader]

The detail I'd anchor on is that this started when the run-as service principal changed. That timing usually means nothing new started referencing the old schema; something has been touching it all along, and the old identity's grants kept it invisible. So the question becomes --- what has this pipeline always been reading or writing there

You can find the exact object without the event log TVF. Unity Catalog permission denials land in the audit system table, so the failing update leaves a trail:

SELECT event_time, action_name,
request_params, response.error_message
FROM system.access.audit
WHERE user_identity.email = '<new SP application id>'
AND response.status_code = 403
ORDER BY event_time DESC

Run that after a failed update and the denied action should name the securable the USE SCHEMA check failed on. That turns the guessing into a lookup.

My prime suspect matches @adnan_alvee 's: the event log. By default a pipeline writes its event log to a hidden Delta table in the catalog and schema configured for the pipeline, and the docs warn that deleting the event log or its parent schema can make future updates fail, which shows how load-bearing it is. If your pipeline was created back when the old schema was the default, that hidden table plausibly still lives there even though every published table moved. databricks pipelines get <pipeline-id> shows the current spec, including an event_log field if one was ever set explicitly.

If the event log is the culprit, there's a cleaner permanent fix than granting the new SP access to the old schema forever. Pipelines support publishing the event log to a location you choose, an event_log block with catalog, schema, and table name in the pipeline settings, so it can live in the new schema with everything else.

Even if you grant USE SCHEMA temporarily, treat it as instrumentation rather than a concession. 

[keshavmonga22]

Hi @binlogreader and @adnan_alvee , thank you for the replies. But I don't have access to system.access either.

We had temporarily granted USE SCHEMA permissions on the older schema to run the pipelines.

Since then, I have deleted the schema entirely since it was anyways inactive now. Please note that when we had changed schemas, we had recreated the tables from scratch in the new schema. 

[binlogreader]

@keshavmonga22 Is there anyone in your team with Admin access that can grant you temporary access or can check the event log?