Currently I am having some issues with the writing of the parquet file in the Storage Container. I do have the codes running but whenever the dataframe writer puts the parquet to the blob storage instead of the parquet file type, it is created as a folder type with many files content to it.
One note to it, I tried searching various ways in the internets that it seems this is default creation when using pyspark and I can see in the folder created there was a file parquet with a snappy add into it (refer to the screenshots below)
If this is a default creation to pyspark code, how can I write a single parquet format that will do some splitting or creation of folder? any recommendations? on how to do it?