DLT pipeline is unable to process struct with hyphen in nested column name

[Rishabh_T]

Hello,

I have some nested columns with hyphen i.e. sample-1 in struct column, recently DLT pipeline has started throwing synatx error. Before May 24, 2023, this was working fine.

Is this a new bug in May 2023 release?

After clearing table and table's data when it's running for first time, then no error is there. But next runs are throwing syntax error.

Note: We are creating live table with SCD-1 using APPLY_CHANGES.

[Kaniz]

Hi @Rishabh Tomar​, Please review the release notes or documentation for the May 2023 release of Delta Lake.

[Rishabh_T]

Thank you for your response @Kaniz Fatma​ .

We have configured DLT pipeline channel to 'current', so it is using DBR 11.3 where it is having Delta Lake 2.1.0. Above mentioned link is for Delta Lake 2.3.0.

 

We have observed that default DBR was updated in DLT as per below to 11.3.11 from 11.3.10 in Release 2023.19.

https://docs.databricks.com/release-notes/delta-live-tables/2023/19/index.html

[Kaniz]

Cool!

So, are you still able to see a similar syntax error?

@Rishabh Tomar​ .

[Rishabh_T]

Yes, syntax error started appearing after May 24, that is when release 2023.19 happened.

[Anonymous]

Hi @Rishabh Tomar​ 

We haven't heard from you since the last response from @Kaniz Fatma​  . Kindly share the information with us, and in return, we will provide you with the necessary solution.

 Thanks and Regards

[Rishabh_T · Accepted solution]

Hi @Vidula Khanna​ ,

Issue got resolved automatically post DLT release 2023.21.

Strangely in release 2023.19, we got this hyphen issue i.e. "syntax error". This was working fine with prior releases also.

If you have more details on this, please do let me know.

Thanks for your support! @Kaniz Fatma​ @Vidula Khanna​ 

[Kaniz]

Hi @Rishabh Kesarwani​, Your input matters! Please help our community thrive by returning and marking the most helpful and accurate answers. Together, we can make a difference!