write to Delta

SSV_dataeng
New Contributor II

‎07-23-2023 08:36 AM

spark.conf.set("spark.databricks.delta.properties.defaults.columnMapping.mode","name")
products_output_path = DA.paths.working_dir + "/delta/products"
products_df.write.format("delta").save(products_output_path)
 
verify_files = dbutils.fs.ls(products_output_path)
verify_delta_format = False
verify_num_data_files = 0
for f in verify_files:
    if f.name == "_delta_log/":
        verify_delta_format = True
    elif f.name.endswith(".parquet"😞
        verify_num_data_files += 1

assert verify_delta_format, "Data not written in Delta format"
assert verify_num_data_files > 0, "No data written"
del verify_files, verify_delta_format, verify_num_data_files
print("All test pass")
 
AssertionError Traceback (most recent call last) <command-1787238488083573> in <cell line: 11>() 9 10 assert verify_delta_format, "Data not written in Delta format" ---> 11 assert verify_num_data_files > 0, "No data written" 12 del verify_files, verify_delta_format, verify_num_data_files 13 print("All test pass") AssertionError: No data written
0 Kudos

daniel_sahal
Databricks MVP

‎07-25-2023 10:50 PM

@SSV_dataeng 
What is the question here?

0 Kudos

SSV_dataeng
New Contributor II
In response to daniel_sahal

‎07-26-2023 03:45 AM

1. I was trying to create a file in Delta format. After Delta File creation at the verification step getting Assertion Error: No data written.

could you please let me know how to fix this issue?

 

0 Kudos

Lakshay
Databricks Employee
Databricks Employee

‎07-26-2023 04:49 AM

I see the error is part of the assertion part of your code. Do you see any other error when writing the data?

0 Kudos

NandiniN
Databricks Employee
Databricks Employee

‎07-27-2023 05:06 AM

Hi @SSV_dataeng ,

Please check with this (you would have to indent it correctly for python)

productsOutputPath = DA.workingDir + "/delta/products"
(productsDF
.write
.format("delta")
.mode("overwrite")
.save(productsOutputPath)
)

verify_files = dbutils.fs.ls(productsOutputPath)
verify_delta_format = False
verify_num_data_files = 0
for f in verify_files:
if f.name == '_delta_log/':
verify_delta_format = True
elif f.name.endswith('.parquet'):
verify_num_data_files += 1

assert verify_delta_format, "Data not written in Delta format"
assert verify_num_data_files > 0, "No data written"
del verify_files, verify_delta_format, verify_num_data_files

0 Kudos