topic Re: Add the creation date of a parquet file into a DataFrame in Data Engineering

Add the creation date of a parquet file into a DataFrame

wyzer — Fri, 18 Nov 2022 16:25:08 GMT

Currently I load multiple parquet file with this code:

df = spark.read.parquet("/mnt/dev/bronze/Voucher/*/*")

(Inside the Voucher folder, there is one folder by date. Each one containing one parquet file)

How can I add a column into this DataFrame, that contains the creation date of each parquet file ?

Thanks

Re: Add the creation date of a parquet file into a DataFrame

MichailKaramano — Fri, 18 Nov 2022 16:43:59 GMT

Hi,

You can use the file metadata column: https://docs.databricks.com/ingestion/file-metadata-column.html

This way you can access the file_path, file_name, file_size and file_modification_time of the data file from the corresponding dataframe row. No need to do it manually!

I found it useful 🙂

Re: Add the creation date of a parquet file into a DataFrame

wyzer — Fri, 18 Nov 2022 20:46:00 GMT

Thanks @Michail Karamanos