Databricks Community

HoussemBL

Dear Community,I have a deployment issue after restructuring my project.Previously, our project was organized with the following structure:project/src/project/resources/project/databricks.ymlAs part of an optimization effort, we have transitioned to ...

HoussemBL · 02-11-2025

Hello community,I am using dlt.apply_changes function to implement SCD2. I am specifying the schema of my streaming_table that should result from apply_changes().This schema contains a generated column.Somehow, my DLT pipeline returns always in first...

HoussemBL · 01-21-2025

Hello community,I have implemented a DLT pipeline.In the "Destination" setting of the pipeline I have specified a unity catalog with target schema of type external referring to an S3 destination.My DLT pipeline works well. Yet, I noticed that all str...

HoussemBL · 12-20-2024

Hello Community,I have a specific scenario regarding Unity Catalog and workspace deletion that I'd like to clarify:Current Setup:Two DataBricks workspaces: W1 and W2Single Unity Catalog instanceCatalog1: Created in W1, shared and accessible in W2Cata...

HoussemBL · 08-14-2024

Hello,I have deployed successfully a Databricks Job that contains one task of type DLT using Databricks Asset Bundle.First deployment works well. For this particular Databricks job, I have clicked on "disconnect from source" to do some customization....

HoussemBL

Hi @ashraf1395 ,I am creating two separate databricks.yml for each sub-project.

HoussemBL · 02-11-2025

Hello @Alberto_Umana Thanks for your reply.I checked the impact of sequencing and ordering. For that, I ran my DLT pipeline with an input dataset of one row. Still, I am getting the same behavior (error for the first run of the DLT pipeline, then suc...

HoussemBL · 01-22-2025

Hello @Alberto_Umana Thanks for your reply.I have followed your proposal. However, I got the following error when launching a DLT pipeline with Unity catalog.java.lang.IllegalArgumentException: Cannot specify an explicit path for a table when using U...

HoussemBL · 11-06-2024

I am facing a similar issue. Is there any solution to select which columns to update when using "apply changes" operation?

Databricks Community

User Stats

User Activity

Databricks asset bundle deploys DLT pipelines as duplicate resources

DLT- apply_changes() SCD2 is not applying defined schema only for first run

External tables in DLT pipelines

Impact of deleting workspace on associated catalogs

Databricks Asset Bundle deploy failure

Re: Databricks asset bundle deploys DLT pipelines as duplicate resources

Re: DLT- apply_changes() SCD2 is not applying defined schema only for first run

Re: External tables in DLT pipelines

Re: DLT Apply Changes