mmayorga
Databricks Employee
Databricks Employee

hi @der 

Thank you for reaching out with your question, very interesting behavior.

I attempted to reproduce your issue using DBR 17.3 LTS but was unable to install the specific library version “dev.mauch:spark-excel_2.13:4.0.0_0.31.2.” However, I installed the next available version, “dev.mauch:spark-excel_2.13:4.0.0_0.32.2-prerelease0.”

With this version, I was able to use the “spark.read.format(“excel”)” function without any issue.

spark.read.format("excel").load(excel_file)

Aside from the links you shared, I couldn’t find documentation explaining why the error occurred with the “Excel” format. It appears that the issue may have been resolved in this newer version, possibly due to a missing internal registration.

I recommend updating your dev.mauch:spark-excel library to the newer version.

Alternatively, you can handle Excel (xlsx) files using the Pandas API on Spark, with the read_excel function, then convert it to a spark dataframe using the code below; just ensure you have “openpyxl” installed (using pip install)

import pyspark.pandas as ps

df = ps.read_excel(excel_file)
sp_df = df.to_spark()
display(sp_df)

 I hope this helps and if it does, please Accept as Solution.
Thank you!