Databricks Community

ksenija

I created my table using SCD type 2 in SQL. I need to do full refresh to load all of the data. Whenever I update data in my source table, in my new table scd_target I see only the latest record, history is not being saved. CREATE OR REFRESH STREAMING...

ksenija

Hello!I created a DLT pipeline where my sources are external tables. I have to apply changes (stored_as_scd_type = 1). However, when I run my pipeline, I don't see any incremental uploads. The data remains in the same state as when I first created th...

ksenija

Hello!I'm trying to do my modeling in DLT pipelines. For bronze, I created 3 streaming views. When I try to join them to create silver table, I got an error that I can't join stream and stream without watermarks. I tried adding them but then I got no...

ksenija · 06-20-2024

Hi!I created DLT pipeline and I'm getting this error:[STREAM_FAILED] Query [id = ***, runId = ***] terminated with exception: object com.databricks.cdc.spark.DebeziumJDBCMicroBatchProvider not found.I'm using Serverless.How to verify that the require...

ksenija · 05-31-2024

Hi everyone,When I tried to create new Databricks job that is using a notebook from a repo, it asked me to set up Azure DevOps Services (Personal access token) in Linked Accounts under my username. And now every time I want to create a new branch or ...

ksenija

Hi @Kaniz_Fatma ,I still don't have a solution for my problem, so I still don't want to close this conversation.Best regards,Ksenija

ksenija

Hi @Rishabh-Pandey ,Those are my catalog, schema and external table

ksenija

Hello @lucasrocha ,Yes, we are receiving new records in our tables and updated_at is always populated. We are using the same script to read data from foreign catalog and that DLT pipeline is working completely fine dlt.apply_changes( targe...

ksenija

Hi Ravi,Thanks! What would you suggest for daily import of data while using DLT pipeline? Using streaming tables with 1 day watermark or to use materialized view?

ksenija

@Dnirmania, @jlachniet I didn’t manage to resolve this issue, but I created a regular notebook and I’m using MERGE statement. If you can’t merge all data at once, you can use a loop with hourly intervals

Databricks Community

User Stats

User Activity

DLT pipeline - SCD type 2

DLT pipeline - reading from external tables

DLT pipeline - silver table, joining streaming data

DLT pipeline - DebeziumJDBCMicroBatchProvider not found

Connecting Databricks and Azure Devops

Re: DLT pipeline - SCD type 2

Re: DLT pipeline - SCD type 2

Re: DLT pipeline - reading from external tables

Re: DLT pipeline - silver table, joining streaming data

Re: DLT pipeline - DebeziumJDBCMicroBatchProvider not found