topic Re: How to make a string column with numeric and alphabet values use as partition? in Data Engineering

How to make a string column with numeric and alphabet values use as partition?

irfanaziz — Wed, 15 Jun 2022 07:44:46 GMT

So i have two partitions defined for this delta table, One is year('GJHAR') contains year values, and the other is a string column('BUKS') with around 124 unique values. However, there is one problem with the 2nd partition column('BUKS'),

The values in BUKS are all numeric(e.g. 151, 017,002) except one('XB'). So if i create the partition by first inserting the 'XB' values followed by the other values (numeric ones ) then it gets inserted. But after the 2nd insert, which did not have any 'XB' values in the column('BUKS'), if we insert or merge a bunch of rows with numeric and 'XB' rows it simply fails with the error :

"cannot resolve XB in search condition given columns base"

I dont understand why XB is considered as a column by Spark?

Re: How to make a string column with numeric and alphabet values use as partition?

-werners- — Fri, 17 Jun 2022 09:08:12 GMT

@nafri A , So to make sure I understand correctly: if you partition the table with only numeric data in BUKS, new incoming data cannot be added if it contains a string; but the other way around it does work?

Could it be that spark has inferred the column as integer/numeric instead of string?

Re: How to make a string column with numeric and alphabet values use as partition?

jose_gonzalez — Fri, 29 Jul 2022 20:05:12 GMT

Hi @nafri A,

Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.