I found what my problem was. I used pandas to save my parquet file to /tmp. It stored it in the compute node local file system /tmp folder. When I passed the same path to pyspark to load the file, it prepended 'dbfs:' to the path. The file wasn't in ...
I am experiencing the same problem. I create a file in /tmp and can verify that it exists. But when an attempt is made to open the file using pyspark, the file is not found. I noticed that the path I used to create the file is /tmp/foobar.parquet and...
