topic Re: DLT pipeline is unable to process struct with hyphen in nested column name in Data Engineering

DLT pipeline is unable to process struct with hyphen in nested column name

Rishabh_T — Tue, 06 Jun 2023 05:53:59 GMT

Hello,

I have some nested columns with hyphen i.e. sample-1 in struct column, recently DLT pipeline has started throwing synatx error. Before May 24, 2023, this was working fine.

Is this a new bug in May 2023 release?

After clearing table and table's data when it's running for first time, then no error is there. But next runs are throwing syntax error.

Note: We are creating live table with SCD-1 using APPLY_CHANGES.

Re: DLT pipeline is unable to process struct with hyphen in nested column name

Rishabh_T — Tue, 06 Jun 2023 14:49:55 GMT

Thank you for your response @Kaniz Fatma .

We have configured DLT pipeline channel to 'current', so it is using DBR 11.3 where it is having Delta Lake 2.1.0. Above mentioned link is for Delta Lake 2.3.0.

We have observed that default DBR was updated in DLT as per below to 11.3.11 from 11.3.10 in Release 2023.19.

https://docs.databricks.com/release-notes/delta-live-tables/2023/19/index.html

Re: DLT pipeline is unable to process struct with hyphen in nested column name

Rishabh_T — Wed, 07 Jun 2023 13:15:12 GMT

Yes, syntax error started appearing after May 24, that is when release 2023.19 happened.

Re: DLT pipeline is unable to process struct with hyphen in nested column name

Anonymous — Wed, 14 Jun 2023 07:06:17 GMT

Hi @Rishabh Tomar

We haven't heard from you since the last response from @Kaniz Fatma . Kindly share the information with us, and in return, we will provide you with the necessary solution.

Thanks and Regards

Re: DLT pipeline is unable to process struct with hyphen in nested column name

Rishabh_T — Wed, 14 Jun 2023 09:34:53 GMT

Hi @Vidula Khanna ,

Issue got resolved automatically post DLT release 2023.21.

Strangely in release 2023.19, we got this hyphen issue i.e. "syntax error". This was working fine with prior releases also.

If you have more details on this, please do let me know.

Thanks for your support! @Kaniz Fatma @Vidula Khanna