cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

DLT pipeline is unable to process struct with hyphen in nested column name

Rishabh_T
New Contributor III

Hello,

I have some nested columns with hyphen i.e. sample-1 in struct column, recently DLT pipeline has started throwing synatx error. Before May 24, 2023, this was working fine.

Is this a new bug in May 2023 release?

After clearing table and table's data when it's running for first time, then no error is there. But next runs are throwing syntax error.

Note: We are creating live table with SCD-1 using APPLY_CHANGES.

Error

1 ACCEPTED SOLUTION

Accepted Solutions

Rishabh_T
New Contributor III

Hi @Vidula Khannaโ€‹ ,

Issue got resolved automatically post DLT release 2023.21.

Strangely in release 2023.19, we got this hyphen issue i.e. "syntax error". This was working fine with prior releases also.

If you have more details on this, please do let me know.

Thanks for your support! @Kaniz Fatmaโ€‹ @Vidula Khannaโ€‹ 

View solution in original post

4 REPLIES 4

Thank you for your response @Kaniz Fatmaโ€‹ .

We have configured DLT pipeline channel to 'current', so it is using DBR 11.3 where it is having Delta Lake 2.1.0. Above mentioned link is for Delta Lake 2.3.0.

DLT DBR 

We have observed that default DBR was updated in DLT as per below to 11.3.11 from 11.3.10 in Release 2023.19.

https://docs.databricks.com/release-notes/delta-live-tables/2023/19/index.html

Rishabh_T
New Contributor III

Yes, syntax error started appearing after May 24, that is when release 2023.19 happened.

Anonymous
Not applicable

Hi @Rishabh Tomarโ€‹ 

We haven't heard from you since the last response from @Kaniz Fatmaโ€‹  . Kindly share the information with us, and in return, we will provide you with the necessary solution.

 Thanks and Regards

Rishabh_T
New Contributor III

Hi @Vidula Khannaโ€‹ ,

Issue got resolved automatically post DLT release 2023.21.

Strangely in release 2023.19, we got this hyphen issue i.e. "syntax error". This was working fine with prior releases also.

If you have more details on this, please do let me know.

Thanks for your support! @Kaniz Fatmaโ€‹ @Vidula Khannaโ€‹ 

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group