cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to make a string column with numeric and alphabet values use as partition?

irfanaziz
Contributor II

So i have two partitions defined for this delta table, One is year('GJHAR') contains year values, and the other is a string column('BUKS') with around 124 unique values. However, there is one problem with the 2nd partition column('BUKS'),

The values in BUKS are all numeric(e.g. 151, 017,002) except one('XB'). So if i create the partition by first inserting the 'XB' values followed by the other values (numeric ones ) then it gets inserted. But after the 2nd insert, which did not have any 'XB' values in the column('BUKS'), if we insert or merge a bunch of rows with numeric and 'XB' rows it simply fails with the error :

"cannot resolve XB in search condition given columns base"

I dont understand why XB is considered as a column by Spark?

2 REPLIES 2

-werners-
Esteemed Contributor III

@nafri A​ , So to make sure I understand correctly: if you partition the table with only numeric data in BUKS, new incoming data cannot be added if it contains a string; but the other way around it does work?

Could it be that spark has inferred the column as integer/numeric instead of string?

Hi @nafri A​,

Just a friendly follow-up. Did any of the responses help you to resolve your question? if it did, please mark it as best. Otherwise, please let us know if you still need help.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group