write to Delta

[SSV_dataeng]

spark.conf.set("spark.databricks.delta.properties.defaults.columnMapping.mode","name")

products_output_path = DA.paths.working_dir + "/delta/products"

products_df.write.format("delta").save(products_output_path)

verify_files = dbutils.fs.ls(products_output_path)

verify_delta_format = False

verify_num_data_files = 0

for f in verify_files:

    if f.name == "_delta_log/":

        verify_delta_format = True

    elif f.name.endswith(".parquet"😞

        verify_num_data_files += 1

assert verify_delta_format, "Data not written in Delta format"

assert verify_num_data_files > 0, "No data written"

del verify_files, verify_delta_format, verify_num_data_files

print("All test pass")

AssertionError Traceback (most recent call last) <command-1787238488083573> in <cell line: 11>() 9 10 assert verify_delta_format, "Data not written in Delta format" ---> 11 assert verify_num_data_files > 0, "No data written" 12 del verify_files, verify_delta_format, verify_num_data_files 13 print("All test pass") AssertionError: No data written

[daniel_sahal]

@SSV_dataeng 
What is the question here?

[SSV_dataeng]

1. I was trying to create a file in Delta format. After Delta File creation at the verification step getting Assertion Error: No data written.

could you please let me know how to fix this issue?

[Lakshay]

I see the error is part of the assertion part of your code. Do you see any other error when writing the data?

[NandiniN]

Hi @SSV_dataeng ,

Please check with this (you would have to indent it correctly for python)

productsOutputPath = DA.workingDir + "/delta/products"
(productsDF
.write
.format("delta")
.mode("overwrite")
.save(productsOutputPath)
)

verify_files = dbutils.fs.ls(productsOutputPath)
verify_delta_format = False
verify_num_data_files = 0
for f in verify_files:
if f.name == '_delta_log/':
verify_delta_format = True
elif f.name.endswith('.parquet'):
verify_num_data_files += 1

assert verify_delta_format, "Data not written in Delta format"
assert verify_num_data_files > 0, "No data written"
del verify_files, verify_delta_format, verify_num_data_files