topic Re: DLT pipeline error key not found: user in Data Engineering

DLT pipeline error key not found: user

ksenija — Mon, 27 May 2024 13:14:50 GMT

When I try to create a DLT pipeline from a foreign catalog (BigQuery), I get this error: java.util.NoSuchElementException: key not found: user.

I used the same script to copy Salesforce data and that worked completely fine.

Re: DLT pipeline error key not found: user

lucasrocha — Mon, 27 May 2024 20:04:44 GMT

Hello @ksenija ,

Are you able to query the data from this foreign catalog out of DLT pipeline?
If so, which channel are you using in your DLT pipeline?
If it is CURRENT channel, could you please try running the pipeline on PREVIEW channel and let me know the results?

Lakehouse query Federation: https://docs.databricks.com/en/query-federation/index.html

Best regards,
Lucas Rocha

Re: DLT pipeline error key not found: user

ksenija — Mon, 27 May 2024 20:13:40 GMT

Hello @lucasrocha,

Yes, I'm able to query my foreign catalog and I'm already using PREVIEW since CURRENT didn't work with Serverless.

Do you know if there is some setup that I'm missing?

Best regards,

Ksenija

Re: DLT pipeline error key not found: user

lucasrocha — Mon, 27 May 2024 21:23:00 GMT

Hey @ksenija ,

Can you share more details? Any source code example?

Best regards,
Lucas Rocha

Re: DLT pipeline error key not found: user

ksenija — Tue, 28 May 2024 07:21:05 GMT

Hello @lucasrocha,

This is part of my code that does DLT:

table = 'my_table'
schema='my_schema'

@dlt.view(name = table + "_tmp_view")
spark.readStream.table(schema + '.' + table)

dlt.create_streaming_table(name = table)
dlt.apply_changes(
target = table,
source = table + "_tmp_view",
keys = ["id"],
sequence_by = col("updated_at"),
stored_as_scd_type = "1"
)

Best regards,

Ksenija