cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Incorrectly truncated long numbers in DS&E workspaces notebooks

Cosimo_F_
Contributor

Hello,

I am getting inconsistent representation of long types.

1661817599972 is the unix timestamp in milliseconds for Monday, August 29, 2022 11:59:59.972 PM GMT

when I execute:

`select 1661817599972 as t`

the result is:

166181759997 (last digit truncated)

The number is correctly cast as a bigint according to `describe select 1661817599972 as t`.

However when I execute:

select timestamp_millis(1661817599972) as t

the result is (correctly) `2022-08-29T23:59:59.972+0000`. So the internal representation of the object seems correct. This appears to happen for bigints only. For instance:

`select 16618175678 as t`

yields:

1661817567

but

`select 1661817567 as t`

also yields

1661817567

DBR 11.2

Thank you,

Cosimo.

PS: displays correctly in DBSQL

8 REPLIES 8

Hubert-Dudek
Esteemed Contributor III

Can you share your code and screenshot? As for me, it worked ok.image.png 

Hi Hubert,

here's the screenshot:

Screen Shot 2022-10-14 at 8.41.50 AM

Executed just now:

Screen Shot 2022-10-14 at 8.48.34 AM

Hubert-Dudek
Esteemed Contributor III

Hi, I tried again on another cluster but still working ok. Maybe try to change the runtime version. Additionally, you can check in Spark UI -> Environment -> Spark properties. Perhaps you have there some options set that change that behavior. I tried to find now in the options something related but couldn't.

Kaniz
Community Manager
Community Manager

Hi @Cosimo Felline​ , We haven’t heard from you since the last response from @Hubert Dudek​ , and I was checking back to see if you have a resolution yet.

If you have any solution, please share it with the community as it can be helpful to others. Otherwise, we will respond with more details and try to help.

Also, Please don't forget to click on the "Select As Best" button whenever the information provided helps resolve your question.

Thank you both for following up, unfortunately I'm still having the issue. Seems like a notebook frontend issue to me...

@Hubert Dudek​ thanks for the suggestion, I will take a look.

Best,

Cosimo.

Cosimo_F_
Contributor

Weirdly, it seems to work if I indicate a long enough column name (or if I specify no column name). This I think supports the hypothesis of a frontend issue.

Screen Shot 2022-10-25 at 7.30.19 PM

Anonymous
Not applicable

Hi @Cosimo Felline​ 

Hope all is well!

Does @Hubert Dudek (Customer)​ response answer your question?

If you could resolve your issue, would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.