cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Robin_200273
by Contributor
  • 11892 Views
  • 11 replies
  • 19 kudos

Resolved! Delta Live Tables failed to launch pipeline cluster

I'm trying to run through the Delta Live Tables quickstart example on Azure Databricks. When trying to start the pipeline I get the following error:Failed to launch pipeline cluster 0408-131049-n3g9vr4r: The operation could not be performed on your a...

  • 11892 Views
  • 11 replies
  • 19 kudos
Latest Reply
kunaldeb
New Contributor III
  • 19 kudos

This communication really helped me. I am now successfully able to execute DLT pipeline. Thanks to all contributor.

  • 19 kudos
10 More Replies
SRK
by Contributor III
  • 1607 Views
  • 5 replies
  • 0 kudos

Delta Live Tables data quality rules application.

I have a requirement, where I need to apply inverse DQ rule on a table to track the invalid data. For which I can use the following approach:import dltrules = {}quarantine_rules = {}rules["valid_website"] = "(Website IS NOT NULL)"rules["valid_locatio...

  • 1607 Views
  • 5 replies
  • 0 kudos
Latest Reply
Hubert-Dudek
Esteemed Contributor III
  • 0 kudos

You can get additional info from DLT event log which is in delta so you can load it as table https://docs.databricks.com/workflows/delta-live-tables/delta-live-tables-event-log.html#data-quality

  • 0 kudos
4 More Replies
knawara
by Contributor
  • 2221 Views
  • 4 replies
  • 1 kudos

Delta Live Tables: reading from output

I'm trying to implement an incremental ingestion logic in the following way:database tables have DbUpdatedDate columnDuring initial load I perform a full copy of the database tableDuring incremental load I:scan the data already in the DLT to see what...

  • 2221 Views
  • 4 replies
  • 1 kudos
Latest Reply
fecavalc08
New Contributor III
  • 1 kudos

Hi @Chris Nawara​, I had the same issue you had. I was trying to avoid the apply_changes but we in the end I implemented it and I'm happier that I expected heheand if you have any additional standardization columns that you need to implement, you can...

  • 1 kudos
3 More Replies
Ender
by New Contributor
  • 565 Views
  • 0 replies
  • 0 kudos

Delta Live Tables migration

How can I migrate a delta live tables workflow to another Databricks workspace?PS: Data source/sink will remain the same. I only want to migrate the DLT config.

  • 565 Views
  • 0 replies
  • 0 kudos
Rishabh264
by Honored Contributor II
  • 1452 Views
  • 6 replies
  • 2 kudos

delta live table

If i have two stages bronze and silver and when i create delta live tables we need to give the target schema to store the results , but i need to store tables in two databases bronze AND silver , for this i need to create two different delta live tab...

  • 1452 Views
  • 6 replies
  • 2 kudos
Latest Reply
Geeta1
Valued Contributor
  • 2 kudos

Hi @Rishabh Pandey​ , yes you have to create 2 DLT tables

  • 2 kudos
5 More Replies
FD_MR
by New Contributor II
  • 772 Views
  • 0 replies
  • 1 kudos

Delta Live Tables executing repeatedly and returning empty DF

Still relatively new to Spark and even more so to Delta Live Tables so apologies if I've missed something fundamental but here goes.We are trying to run a notebook via Delta Live Tables, which contains 2 functions decorated by the `dlt.table` decorat...

  • 772 Views
  • 0 replies
  • 1 kudos
repcak
by New Contributor III
  • 3037 Views
  • 6 replies
  • 3 kudos

Resolved! Delta Live Tables with EventHub

Hello,I would like to integrate Databricks Delta Live Tables with Eventhub, but i cannot install com.microsoft.azure:azure-eventhubs-spark_2.12:2.3.17 on delta live cluster.I tried installed in using Init script (by adding it in Json cluster settings...

image
  • 3037 Views
  • 6 replies
  • 3 kudos
Latest Reply
Atanu
Esteemed Contributor
  • 3 kudos

I think this has some details https://docs.microsoft.com/en-us/azure/event-hubs/event-hubs-kafka-spark-tutorial @Kacper Mucha​ is the issue resolved ?

  • 3 kudos
5 More Replies
SM
by New Contributor III
  • 4763 Views
  • 8 replies
  • 3 kudos

Resolved! Delta Live Tables has duplicates created by multiple workers

Hello, I am working with Delta Live Tables, I am trying to create a DLT from a combination of Dataframes from a 'for loop' which are unioned and then DLT is created over the Unioned Dataframe. However I noticed that the delta table has duplciates. An...

  • 4763 Views
  • 8 replies
  • 3 kudos
Latest Reply
Anonymous
Not applicable
  • 3 kudos

@Shikha Mathew​ - Does your last answer mean that your issue is resolved? Would you be happy to mark whichever answer helped as best? Or, if it wasn't a specific one, would you tell us what worked?

  • 3 kudos
7 More Replies
Labels