topic Write Empty Delta file in Datalake in Data Engineering

Write Empty Delta file in Datalake

BhagS — Tue, 17 May 2022 13:36:03 GMT

hi all,

Currently, i am trying to write an empty delta file in data lake, to do this i am doing the following:

Reading parquet file from my landing zone ( this file consists only of the schema of SQL tables)

df=spark.read.format('parquet').load(landingZonePath)

After this, i convert this file into the delta

df.write.format("delta").save(centralizedZonePath)

But after checking data lake i see no file

Note: Parquet file in landingzone, has the schema

Re: Write Empty Delta file in Datalake

Hubert-Dudek — Tue, 17 May 2022 16:52:48 GMT

@bhagya s , File schema is in _delta_log

Re: Write Empty Delta file in Datalake

Noopur_Nigam — Thu, 02 Jun 2022 03:20:42 GMT

Hi @bhagya s Since your source file is empty, there is no data file inside the centralizedZonePath directory i.e .parquet file is not created in the target location. However, _delta_log is the transaction log that holds the metadata of the delta format data and has the schema of the table.

You may understand more about transaction logs here :https://databricks.com/discover/diving-into-delta-lake-talks/unpacking-transaction-log