cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Delta file partitions

thushar
Contributor

Have one function to create files with partitions, in that the partitions are created based on metadata (getPartitionColumns) that we are keeping. In a table we have two columns that are mentioned as partition columns, say 'Team' and 'Speciality'.

While executing, partition columns are not substituted properly within the datafrme's write method and getting an error like below

AnalysisException: Partition column `"Team","Speciality"` not found in schema

But these columns are already there in the data frame. Any idea how to resolve this?

Seems like the value `"Team","Speciality" is considered as single column instead of separate columns.

def dfWrite(df, targetPath,tableName):

partitionColumn = getPartitionColumns(tableName)

# "Team", "Speciality"

df.write.option("header", True) \

        .partitionBy(partitionColumn) \

        .mode("overwrite") \

        .csv(targetPath)

4 REPLIES 4

pvignesh92
Honored Contributor

Hi Thushar,

You have not mentioned the return type of the getPartitionColumns method. You have to return the partition columns as collection Ex list ['Team', 'Speciality']

Then the below method should work.

df.write.option("header", True) \

        .partitionBy(*partitionColumn) \

        .mode("overwrite") \

        .csv(targetPath)

Kindly try.

Hi Vignesh,

Thanks, the return type was a string, and converted that to a tuple and it is working.

pvignesh92
Honored Contributor

Hi Thushar,

Please upvote and mark this as answer so that the thread will be closed

Anonymous
Not applicable

Hi @Thushar R​ 

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you. 

Cheers!

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.