Thanks Mich, you are partially right and it helped a lot!
Using your code, I was able to see, that it also wrote files with ".c000.csv" at the end. https://stackoverflow.com/questions/54190082/spark-structured-streaming-producing-c000-csv-files say...
The check, if the file is available must use
if len(dbutils.fs.ls(success_file)) > 0:
print("_SUCCESS file found (might not reflect reality if error occurred earlier)")
else:
print("_SUCCESS file missing (indicates incomplete write)")
though (os checks the local file system of the master node, no?)
And even though the files end with ".c000.csv" , I was able to read them in:
test_df = spark.read.option("basePath", "/tmp/output").csv("/tmp/output/", header=True)
And even though the files end with ".c000.csv" , I was able to read them in:
test_df = spark.read.option("basePath", "/tmp/output").csv("/tmp/output/", header=True)
test_df.show()
