Databricks Community

ismaelhenzel

Regarding the limitations of declarative pipelines—specifically the inability to use replaceWhere—I discovered through testing that materialized views actually support dynamic overwrites. This handles several scenarios where replaceWhere would typica...

ismaelhenzel

I would like to know the best practice for collaborating on a Delta Live Tables pipeline. I was thinking that each developer should have their own DLT pipeline in the development workspace. Currently, each domain has its development catalog, like sal...

ismaelhenzel

I would like to understand the differences between Terraform and Asset Bundles, especially since in some cases, they can do the same thing. I’m not talking about provisioning storage, networking, or the Databricks workspace itself—I know that is Terr...

ismaelhenzel

I am using Databricks Apps, and I want to grant privileges to the auto-generated service principal from the app to read specific Unity Catalog schemas. I can't find a way to do this automatically using Asset Bundles.I prefer to query using a service ...

ismaelhenzel · 08-25-2025

My team is migrating pipelines from Spark to Delta Live Tables (DLT), but we've found that some important features, like schema evolution for tables with enforced schemas, seem to be missing. In DLT, we can define schemas, set primary and foreign key...

ismaelhenzel

I totally agree. It’s a mess that DLT can't handle self-referential constraints. I’ve had to create some ugly workaround functions to get past this—like a function that tries to read the referenced table and returns an empty string if it throws an er...

ismaelhenzel

Thanks for answering! That was very clarifying. I've never used it that way before. At first glance, it seems a little strange, but as you said, what matters is QA/Production.

ismaelhenzel

That's an elegant solution; I can understand it better now. I developed my version slightly differently: instead of then set the schema for my DLT pipeline. I used the configuration variable DLT_GOLD_SCHEMA for the target database, as shown below.tar...

ismaelhenzel

That is a very clear explanation; I understand it well now. Bundles seem better for simplicity and a repo-oriented workflow, making it easy to manage multiple repos where everyone creates their own Databricks resources. However, when I need stricter ...

ismaelhenzel

This is a possible solution. The Databricks service principal for the data app only exists after its deployment in the platform. Then, after deployment, I could get the ID that the principal of the data app received, and then apply the necessary read...

Databricks Community

User Stats

User Activity

Declarative Pipelines - Dynamic Overwrite

delta live tables - collaborative development

Asset bundle vs terraform

Databricks Apps - Automating Unity Catalog Privileges for Databricks Apps Service Principals

Schema Evolution/Type Widening in Materialized Views

Re: Self-referential foreign key constraint for streaming tables

Re: delta live tables - collaborative development

Re: delta live tables - collaborative development

Re: Asset bundle vs terraform

Re: Databricks Apps - Automating Unity Catalog Privileges for Databricks Apps Service Principals