- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
10-27-2025 01:54 AM
We ran into similar issues with our client, while migrating their on-prem table relationships to Databricks via DLT/LDP.
The first proposed solution does not work on Materialized Views (MV).
> 'ALTER TABLE ... ADD CONSTRAINT' expects a table but <object> is a view.
Whereas ALTER VIEW or MATERIALIZED VIEW do not support ADD CONSTRAINT
Your second idea with constructing the schema dynamically still runs into the issue of the original poster, where referenced MVs may not exist yet, since they may be constructed later in the pipeline.
The third option of explicitly enforcing the table creation sequence may work, but for large pipelines with hundreds of MVs it becomes a nightmare to manage and lowers the Pipeline performance significantly. Additionally, it does not work if a pair of referenced tables is created in two separate Pipelines.
It also potentially leads to errors silently slipping up through the deployment process, since on dev/test environments, the pipelines may have tables populated from before (=no errors on FK setting), but after doing a fresh pipeline deployment (i.e. in prod), the errors will manifest.
Please add ADD CONSTRAINT to ALTER MATERIALIZED VIEW, so that we can safely batch-update all Materialized Views after the Pipeline(s) have finished.
Alternatively, if foreign keys are set mid-Pipeline via table schemas, defer the reference validation until the Pipeline has finished (still suboptimal solution, since it may not work if the MVs are created in separate parallel Pipelines under the same job).
P.S. Unrelated, but please consider supporting PK and FK to be set via the StructType schema format, not only the string-based schema.
Thank you!