Write Empty Delta file in Datalake

BhagS
New Contributor II

‎05-17-2022 06:36 AM

hi all,

Currently, i am trying to write an empty delta file in data lake, to do this i am doing the following:

  • Reading parquet file from my landing zone ( this file consists only of the schema of SQL tables)
df=spark.read.format('parquet').load(landingZonePath)
  • After this, i convert this file into the delta
df.write.format("delta").save(centralizedZonePath)
  • But after checking data lake i see no file
  • image

Note: Parquet file in landingzone, has the schema

Preview file
34 KB

Hubert-Dudek
Databricks MVP

‎05-17-2022 09:52 AM

@bhagya s​ , File schema is in _delta_log


My blog: https://databrickster.medium.com/

Noopur_Nigam
Databricks Employee
Databricks Employee

‎06-01-2022 08:20 PM

Hi @bhagya s​ Since your source file is empty, there is no data file inside the centralizedZonePath directory i.e .parquet file is not created in the target location. However, _delta_log is the transaction log that holds the metadata of the delta format data and has the schema of the table.

You may understand more about transaction logs here :https://databricks.com/discover/diving-into-delta-lake-talks/unpacking-transaction-log

View solution in original post