topic Re: Delta file partitions in Data Engineering

Delta file partitions

thushar — Thu, 09 Mar 2023 07:57:42 GMT

Have one function to create files with partitions, in that the partitions are created based on metadata (getPartitionColumns) that we are keeping. In a table we have two columns that are mentioned as partition columns, say 'Team' and 'Speciality'.

While executing, partition columns are not substituted properly within the datafrme's write method and getting an error like below

AnalysisException: Partition column `"Team","Speciality"` not found in schema

But these columns are already there in the data frame. Any idea how to resolve this?

Seems like the value `"Team","Speciality" is considered as single column instead of separate columns.

def dfWrite(df, targetPath,tableName):

partitionColumn = getPartitionColumns(tableName)

# "Team", "Speciality"

df.write.option("header", True) \

.partitionBy(partitionColumn) \

.mode("overwrite") \

.csv(targetPath)

Re: Delta file partitions

pvignesh92 — Thu, 09 Mar 2023 09:54:57 GMT

Hi Thushar,

You have not mentioned the return type of the getPartitionColumns method. You have to return the partition columns as collection Ex list ['Team', 'Speciality']

Then the below method should work.

df.write.option("header", True) \

.partitionBy(*partitionColumn) \

.mode("overwrite") \

.csv(targetPath)

Kindly try.

Re: Delta file partitions

thushar — Thu, 09 Mar 2023 11:05:36 GMT

Hi Vignesh,

Thanks, the return type was a string, and converted that to a tuple and it is working.

Re: Delta file partitions

pvignesh92 — Thu, 09 Mar 2023 11:08:16 GMT

Hi Thushar,

Please upvote and mark this as answer so that the thread will be closed

Re: Delta file partitions

Anonymous — Sat, 01 Apr 2023 00:52:51 GMT

Hi @Thushar R

Hope everything is going great.

Just wanted to check in if you were able to resolve your issue. If yes, would you be happy to mark an answer as best so that other members can find the solution more quickly? If not, please tell us so we can help you.

Cheers!