topic Re: write to Delta in Get Started Discussions

write to Delta

SSV_dataeng — Sun, 23 Jul 2023 15:36:24 GMT

spark.conf.set("spark.databricks.delta.properties.defaults.columnMapping.mode","name")

products_output_path = DA.paths.working_dir + "/delta/products"

products_df.write.format("delta").save(products_output_path)

verify_files = dbutils.fs.ls(products_output_path)

verify_delta_format = False

verify_num_data_files = 0

for f in verify_files:

if f.name == "_delta_log/":

verify_delta_format = True

elif f.name.endswith(".parquet"😞

verify_num_data_files += 1

assert verify_delta_format, "Data not written in Delta format"

assert verify_num_data_files > 0, "No data written"

del verify_files, verify_delta_format, verify_num_data_files

print("All test pass")

AssertionError Traceback (most recent call last) <command-1787238488083573> in <cell line: 11>() 9 10 assert verify_delta_format, "Data not written in Delta format" ---> 11 assert verify_num_data_files > 0, "No data written" 12 del verify_files, verify_delta_format, verify_num_data_files 13 print("All test pass") AssertionError: No data written

Re: write to Delta

daniel_sahal — Wed, 26 Jul 2023 05:50:26 GMT

@SSV_dataeng
What is the question here?

Re: write to Delta

SSV_dataeng — Wed, 26 Jul 2023 10:45:02 GMT

1. I was trying to create a file in Delta format. After Delta File creation at the verification step getting Assertion Error: No data written.

could you please let me know how to fix this issue?

Re: write to Delta

Lakshay — Wed, 26 Jul 2023 11:49:47 GMT

I see the error is part of the assertion part of your code. Do you see any other error when writing the data?

Re: write to Delta

NandiniN — Thu, 27 Jul 2023 12:06:30 GMT

Hi @SSV_dataeng ,

Please check with this (you would have to indent it correctly for python)

productsOutputPath = DA.workingDir + "/delta/products"
(productsDF
.write
.format("delta")
.mode("overwrite")
.save(productsOutputPath)
)

verify_files = dbutils.fs.ls(productsOutputPath)
verify_delta_format = False
verify_num_data_files = 0
for f in verify_files:
if f.name == '_delta_log/':
verify_delta_format = True
elif f.name.endswith('.parquet'):
verify_num_data_files += 1

assert verify_delta_format, "Data not written in Delta format"
assert verify_num_data_files > 0, "No data written"
del verify_files, verify_delta_format, verify_num_data_files