topic Re: parquet file to include partitioned column in file in Data Engineering

parquet file to include partitioned column in file

guruv — Tue, 28 Dec 2021 06:22:53 GMT

HI,

I have a daily scheduled job which processes the data and write as parquet file in a specific folder structure like root_folder/{CountryCode}/parquetfiles. Where each day job will write new data for countrycode under the folder for countrycode

I am trying to achieve this by using

dataframe.partitionBy("countryCode").write.parquet(root_Folder)

this is creation a folder structure like

root_folder/countryCode=x/part1-snappy.parquet

root_folder/countryCode=x/part2-snappy.parquet

root_folder/countryCode=y/part1-snappy.parquet

but the coutryCode column is removed from the parquet file.

In my case the parquet file is to be read by external consumers and they expect the coutryCode column in file.

Is there an option to have the column in the file and also in folder path.

Re: parquet file to include partitioned column in file

Hubert-Dudek — Tue, 28 Dec 2021 10:45:32 GMT

Most external consumers will read partition as column when are properly configured (for example Azure Data Factory or Power BI).

Only way around is that you will duplicate column with other name (you can not have the same name as it will generate conflicts in appends and reads from many clients):

.withColumn("foo_", col("foo"))

Re: parquet file to include partitioned column in file

guruv — Tue, 28 Dec 2021 13:05:57 GMT

Thanks for reply. Can you suggest consumers when reading custom code to read files can get partitional column?

presently consumer is getting list of all files in folder, and filtering out files already processed and then read each new file with

spark.read.format('parquet').load(filePath)

Re: parquet file to include partitioned column in file

Hubert-Dudek — Wed, 29 Dec 2021 09:58:09 GMT

please try to add .option("mergeSchema", "true")
in filePath just specify main top folder with partitions (root folder for parquet dataset)

Here is official documentation about partition discovery https://spark.apache.org/docs/2.3.1/sql-programming-guide.html#partition-discovery

Re: parquet file to include partitioned column in file

guruv — Wed, 29 Dec 2021 19:28:53 GMT

Thanks will try and check back in case of any other issue.