cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Mado
by Valued Contributor II
  • 1912 Views
  • 4 replies
  • 0 kudos

Medallion architecture, how to update Gold tables?

Assume that I have a data source that is ingested to a few bronze tables, and transformed to a silver table. Ans next, a gold table is created by aggregating the silver table. If new records arrive in the data source, bronze and silver tables are upd...

  • 1912 Views
  • 4 replies
  • 0 kudos
Latest Reply
Mado
Valued Contributor II
  • 0 kudos

Hi @Vidula Khanna​ The answer didn't fit my question. In the case of using Merge, I found a good article here:https://medium.com/@avnishjain22/simplify-optimise-and-improve-your-data-pipelines-with-incremental-etl-on-the-lakehouse-61b279afadea

  • 0 kudos
3 More Replies
JRT5933
by New Contributor III
  • 1275 Views
  • 4 replies
  • 6 kudos

Resolved! GOLD table slowed down at MERGE INTO

Howdy - I recently took a table FACT_TENDER and made it into a medalliona tyle TABLE to test performance since I suspected medallion would be quicker. Key differences: Both tables use bronze dataoriginal has all logic in one long notebookMERGE INTO t...

  • 1275 Views
  • 4 replies
  • 6 kudos
Latest Reply
JRT5933
New Contributor III
  • 6 kudos

I ended up instituing true and tried PARTITIONING and PRUNING methods to boost performance, which has succeeded.

  • 6 kudos
3 More Replies
Raghu_Bindingan
by New Contributor III
  • 1829 Views
  • 4 replies
  • 2 kudos

Truncate delta live table and try to repopulate it in the pipeline

Has anyone attempted to truncate a delta live gold level table that gets populated via a pipeline and then tried to repopulate it by starting the pipeline. I have this situation wherein i need to reprocess all data in my gold table, so i stopped the ...

  • 1829 Views
  • 4 replies
  • 2 kudos
Latest Reply
Rajeev45
New Contributor III
  • 2 kudos

Please can you confirm if the job is still failing with the same error even after “FULL REFRESH ALL” option? If so please share the full stack trace and is it failing in any of the below steps?Creating updateWaiting for resourcesInitializingResetting...

  • 2 kudos
3 More Replies
Labels