Hello @ksenija ,

Are you able to query the data from this foreign catalog out of DLT pipeline?
If so, which channel are you using in your DLT pipeline?
If it is CURRENT channel, could you please try running the pipeline on PREVIEW channel and let me know the results?

Lakehouse query Federation: https://docs.databricks.com/en/query-federation/index.html

Best regards,
Lucas Rocha

Hello @lucasrocha,

Yes, I'm able to query my foreign catalog and I'm already using PREVIEW since CURRENT didn't work with Serverless.

Do you know if there is some setup that I'm missing?

Best regards,

Ksenija

Hey @ksenija ,

Can you share more details? Any source code example?

Best regards,
Lucas Rocha

Hello @lucasrocha,

This is part of my code that does DLT:

table = 'my_table'
schema='my_schema'

@dlt.view(name = table + "_tmp_view")
spark.readStream.table(schema + '.' + table)

dlt.create_streaming_table(name = table)
dlt.apply_changes(
target = table,
source = table + "_tmp_view",
keys = ["id"],
sequence_by = col("updated_at"),
stored_as_scd_type = "1"
)

Best regards,

Ksenija

Hello @lucasrocha ,

Did you have time to check my code?

Thanks in advance,

Ksenija

Hey @ksenija , 

Could you please share the full error stack trace so that we can further check?

Best regards,
Lucas Rocha

Hi @lucasrocha ,

Here is the whole error.

Best regards,

Ksenija

Hi @lucasrocha ,

Any luck with this error? I guess it's something with connection to BigQuery, but I didn't find anything.

Best regards,

Ksenija