topic Re: Performance issues when loading an Excel file from DBFS using R in Data Engineering

Performance issues when loading an Excel file from DBFS using R

mbaumga — Tue, 18 Apr 2023 10:34:20 GMT

I have uploaded small Excel files on my DBFS. I then use function read_xlsx() from the "readxl" package in R to import the file into the R memory. I use a standard cluster (12.1, non ML). The function works but it takes ages. E.g. a simple Excel table with 40000+ records and 5 columns takes 9 minutes. On my R installation on Windows, the load is instantaneous. "readxl" is considered to be the best package to deal with Excel files. It is part of the already made available libraries on the cluster. Any idea what might cause this?

Re: Performance issues when loading an Excel file from DBFS using R

mbaumga — Thu, 20 Apr 2023 05:50:07 GMT

Dear Kaniz, thanks. Your suggestion to copy the Excel file from DBFS to a temp folder "e.g. /tmp/" worked. Once I run read_xlsx() the upload is instantenous.

Indeed, we had the same problem the other way around. I was not able to save an xlsx or pdf file into a folder on "/dbfs/FileStore/*". However, I can save it to "/tmp/*" and then copy it to where I want the file to be. Good to know!

Thanks again!

Re: Performance issues when loading an Excel file from DBFS using R

Anonymous — Mon, 24 Apr 2023 04:29:01 GMT

Hi @Marcel Baumgartner

Thank you for posting your question in our community! We are happy to assist you.

To help us provide you with the most accurate information, could you please take a moment to review the responses and select the one that best answers your question?

This will also help other community members who may have similar questions in the future. Thank you for your participation and let us know if you need any further assistance!

Re: Performance issues when loading an Excel file from DBFS using R

mbaumga — Fri, 28 Apr 2023 10:55:50 GMT

The questions was answered, successfully. All okay!