Compute kind SERVERLESS_REPL_VM is not allowed to use cluster scoped libraries.

[ManojkMohan]

i have a s3 uri 's3://salesforcedatabricksorders/orders_data.xlsx' , i have created a connector between data bricks and salesfoce, i am first gettig the orders_data.xlsx to databricks layer perform basic transformation on it and then send it to salesforce data cloud 

Following the below steps but getting the error int he screen shot , i am using databricks free version

1. Navigate to your cluster's configuration page in the Databricks UI.
2. Go to the Libraries tab.
3. Click Install New.
4. Select Maven as the source.
5. In the Coordinates box, enter: com.crealytics:spark-excel_2.12:3.5.0
6. Click Install. The cluster will restart with the new library. @szymon_dybczak  have you come across this before ?

[szymon_dybczak · Accepted solution]

Hi @ManojkMohan ,

This is a well known limitation of serverless compute. Moreover, serverless doesn't support installing libraries at cluster scope. Only notebook scoped libraries are supported. 

In your case, as a workaround you can try to use pandas to read excel

https://docs.databricks.com/aws/en/compute/serverless/limitations

[kerem]

Hello,

I’ve come across the same issue reading an Excel file into a PySpark dataframe via Serverless compute. 
As the error states with Serverless, you cannot install a cluster scoped library so you have to use notebook scoped libraries (%pip install…). Unfortunately that doesn’t apply to Maven libraries which is annoying. 

As a workaround you can use Pandas to read the Excel file first and read it into a PySpark Dataframe. 

Kerem Durak