Databricks Community

Manjula_Ganesap · 06-13-2024

Hi All, I'm trying to change our Ingestion process to use Autoloader to identify new files landing in a directory on ADLS. The ADLS directory has access tier enabled to archive files older than a certain time period. When I'm trying to set up Autoloa...

Manjula_Ganesap · 09-06-2023

With no change in code, i've noticed that my DLT initialization fails and then an automatic rerun succeeds. Can someone help me understand this behavior. Thank you.

Manjula_Ganesap · 09-06-2023

I have a DLT code to create 40+ bronze tables. The tables are created on top of the latest parquet files for each of those tables. While executing the pipeline, sometimes I notice that the graph is different than the regular one i see. I do not under...

Manjula_Ganesap · 08-18-2023

Hi All,I set up a DLT pipeline to create 58 bronze tables and a subsequent DLT live table that joins the 58 bronze tables created in the first step. The pipeline runs successfully most times.My issue is that the pipeline fails once every 3/4 runs say...

Manjula_Ganesap · 09-21-2023

@jose_gonzalez @Kaniz_Fatma - Missed to update the group on the fix. Reached out to Databricks to understand and it was identified that the threads call that i was making was causing the issue. After i removed it - i don't see it happening.

Manjula_Ganesap · 09-21-2023

@jose_gonzalez - Missed to update the group on the fix. Reached out to Databricks to understand and it was identified that the threads call that i was making was causing the issue. After i removed it - i don't see it happening.

Manjula_Ganesap · 09-06-2023

@Kaniz_Fatma - Thank you for your response. There is no change in the table dependencies. The code to create the individual raw tables look like this: The input to this is always the same 40 tables with only the underlying parquet file changing. I c...

Manjula_Ganesap · 08-24-2023

@Mo - it worked. Thank you so much.

Manjula_Ganesap · 08-24-2023

@Mo - Thank you. I will try this out now and keep you posted. I am directly passing the parameters in the body of the Web Activity with the POST command. I will try the update and let you know. Thank you so much for the suggestion.

Databricks Community

User Stats

User Activity

Autoloader on ADLS blobs with archival enabled

Delta Live Table (DLT) Initialization fails frequently

Delta Live Table Graph different with no change in Notebook code

Delta Live Table pipeline failure - Table missing

Re: Delta Live Table pipeline failure - Table missing

Re: Delta Live Table (DLT) Initialization fails frequently

Re: Delta Live Table Graph different with no change in Notebook code

Re: How to pass configuration values to a Delta Live Tables job through the Delta Live Tables API

Re: How to pass configuration values to a Delta Live Tables job through the Delta Live Tables API