hi @dsnde49 ,

When using Python instead of Spark in Databricks, the data you write will be stored in the drivers local storage.

To avoid this, you can utilize the spark-excel jar from Crealytics (Maven Repository: com.crealytics » spark-excel). This tool allows you to read data from individual sheets in an Excel file and combine all sheet dataframes into one dataframe using either Join or Union, depending on the nature of the data. Finally, you can collate the dataframe into a single partition using df.coalesce(1).

You can follow this medium post to refer how to use spark-excel library  How to Read Excel files using pyspark in Databricks? | by Amit Joshi | Medium

Regards,
Hari Prasad