topic Re: Experiencing sorting problems with bigint columns in Data Engineering

Experiencing sorting problems with bigint columns

Parth2692 — Thu, 12 Jun 2025 04:31:38 GMT

Experiencing sorting problems with bigint columns across tables tested.

Example:

In the table the projectid (bigint as per schema - which is correct) yet when sorted in SQL is sorting as per a string value

1000903 is returned as the minimum projectid value

998401 is returned as the maximum projectid value.

Re: Experiencing sorting problems with bigint columns

EktaPuri — Thu, 12 Jun 2025 05:47:40 GMT

Can you send across schema of table using describe command.

Also at runtime cast it to bigint and then check.

Re: Experiencing sorting problems with bigint columns

Parth2692 — Thu, 12 Jun 2025 05:55:07 GMT

Re: Experiencing sorting problems with bigint columns

Parth2692 — Thu, 12 Jun 2025 05:55:55 GMT

Re: Experiencing sorting problems with bigint columns

EktaPuri — Thu, 12 Jun 2025 06:03:29 GMT

Can you cast explicitly to bigint and check, it's kinda strange

Re: Experiencing sorting problems with bigint columns

Parth2692 — Thu, 12 Jun 2025 06:12:04 GMT

When I use CAST, it returns the data type as BIGINT, which is expected. However, if I'm reading directly from the Unity Catalog (UC) table, it should already return the BIGINT data type without requiring a cast. Is there a reason why this inconsistency is occurring?

Re: Experiencing sorting problems with bigint columns

EktaPuri — Thu, 12 Jun 2025 07:05:36 GMT

Try running explain command what's happening in the background, also if table is small try to write in another table and check