topic Unable to locate saved data in Data Engineering

Unable to locate saved data

dsnde49 — Fri, 10 Jan 2025 06:22:02 GMT

Hi,

I have been trying to save some processed data using pandas from my databricks notebook.

I have tried two versions, using csv and xlsx. The code for both of them runs without any error, but I'm unable to find the location of the saved data.

for table, df in table_schemas.items():

df.to_csv(f"{table}.csv", index=False)

excel_file_path = "https://xyz/file.xlsx"

with pd.ExcelWriter(excel_file_path) as writer:

for table, df in table_schemas.items():

df.to_excel(writer, sheet_name=table, index=False)

I know how to save using spark, but trying out this functionality as i need data for multiple sheets and combine them into one excel. any suggestion is appreciated.

Thanks!

Re: Unable to locate saved data

hari-prasad — Fri, 10 Jan 2025 07:11:41 GMT

hi @dsnde49 ,

When using Python instead of Spark in Databricks, the data you write will be stored in the drivers local storage.

To avoid this, you can utilize the spark-excel jar from Crealytics (Maven Repository: com.crealytics » spark-excel). This tool allows you to read data from individual sheets in an Excel file and combine all sheet dataframes into one dataframe using either Join or Union, depending on the nature of the data. Finally, you can collate the dataframe into a single partition using df.coalesce(1).

You can follow this medium post to refer how to use spark-excel library How to Read Excel files using pyspark in Databricks? | by Amit Joshi | Medium

Regards,
Hari Prasad