I found what my problem was. I used pandas to save my parquet file to /tmp. It stored it in the compute node local file system /tmp folder. When I passed the same path to pyspark to load the file, it prepended 'dbfs:' to the path. The file wasn't in dbfs:/tmp, so the call failed. I prepended 'file:' to the path name that I passed to pyspark and the call succeeded.