Databricks Community

Anish_2

Hello Team,My intialization stage in lakehouse pipeline is taking 5-6 mins. When i checked event log table,below are statsMessage Reported flow time metrics for flowName: 'pipelines.flowTimeMetrics.missingFlowName'. is repeatedly coming for 4-5 mins....

Anish_2 · 01-13-2026

Hello Team,I have use-case in which i want to trigger another dlt pipeline if 1 table got succeded in my parent dlt pipeline. I dont want to create pipeline to pipeline dependency. Is there any way to create table to pipeline dependency?Thank youAnis...

Anish_2 · 04-08-2025

Hello Team,I have scenario where in apply_changes, i want to ignore updates on 1 column. Is there any way we can achieve this in Delta live tables?

Anish_2 · 02-21-2025

Hello Team,I have removed definition of table from delta live table pipeline but table is still present in unity catalog. In event log, it is giving below messageMaterialized View '`catalog1`.`schema1`.`table1`' is no longer defined in the pipeline a...

Anish_2 · 10-04-2024

Hello Team,In order to avoid any cost of running workflow for historical data, i am trying DEEP clone in order to copy data of streaming table from PROD workspace to QA workspace but it is giving below error. Please suggest some efficient data copy m...

Anish_2 · 03-22-2026

@Advika Currently, if we put run_if: ALL_DONE on task level, it is flagged as succeededwithfailures. But there is another workflow under which this workflow runs. In master workflow, it is marked as succeeded only not succeededwithfailures.My orches...

Anish_2 · 03-20-2026

@SteveOstrowski i have encountered one more issue with autocdc. Below is snipped of my autocdc where there are 2 source tables for vlc_hist. After completion, i validated data but data is missing from source 1 is still missiing from vlc_hist silver ...

Anish_2 · 03-16-2026

Hello @SteveOstrowski currently there are 2 pipelines running in parallel. Both pipeline consists of bronze,view and silver table. 1st pipeline is having around 40+ tables and 2nd has 20. I am not able to debug why pipeline with less table is taking...

Anish_2 · 03-16-2026

Hello @SteveOstrowski Thank you for detailed explanation for why partition pruning don't work in streaming queries. Currently I am using dlt.view instead of dlt.table. Will @dlt.view also works as same as dlt.table? As I want to avoid creation of dup...

Anish_2 · 02-16-2026

Basically here ask is to read bronze table new files only using autoloader then do some basic transformation like renaming of columns then do autocdc in silver. But when we do readsteam on bronze, it scan whole table instead of partition pruning. Can...

Databricks Community

User Stats

User Activity

Intialization stage is taking time in lakehouse pipeline

daabricks workflow design

Delta live tables - ignore updates on some columns

removal of Delta live tables

Unable to clone delta live table

Re: should have the option to mark succeeded with failures as a failure rather than a success

Re: DLT table reading not performing file pruning on partition column

Re: DLT table reading not performing file pruning on partition column

Re: DLT table reading not performing file pruning on partition column

Re: DLT table reading not performing file pruning on partition column