py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.feature.VectorAssembler(java.lang.String) is not whitelisted.

[DataRabbit]

Hello, i have a problem.

When I try to run the MLlib Assembler (from pyspark.ml.feature import VectorAssembler) I get this error and I don't know what to do anymore. Please help.

[Hubert-Dudek · Accepted solution]

Is it High Concurrency cluster with credential passthrough enabled? In that case, you can use a different cluster mode.

https://docs.azuredatabricks.net/spark/latest/data-sources/azure/adls-passthrough.html

This exception is thrown when you have accessed a method that Azure Databricks has not explicitly marked as safe for Azure Data Lake Storage credential passthrough clusters. In most cases, the technique could allow a user on an Azure Data Lake Storage credential passthrough cluster to access another user’s credential.

[DataRabbit]

Thanks you so much!

[DBXC]

How could we whitelist this error below with DBR 13.3 and above? 

Py4JError: An error occurred while calling None.org.apache.spark.ml.recommendation.ALS. Trace: py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.recommendation.ALS(java.lang.String) is not whitelisted. at py4j.security.WhitelistingPy4JSecurityManager.checkConstructor(WhitelistingPy4JSecurityManager.java:451) at py4j.Gateway.invoke(Gateway.java:256) at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:199) at py4j.ClientServerConnection.run(ClientServerConnection.java:119) at java.lang.Thread.run(Thread.java:750)

[SudiptaDe]

I am facing the same error with Databricks Serverless (Free Edition) .... please fix it

[VenuG]

Do you plan to support this in Serverless Free Edition? Migration from Community Edition to Serveless has been fraught with these limitations.

[rabbit-task]

I encountered the same issue after switching from Databricks Community Edition to Databricks Free Edition. After I detach and reattach the serverless compute, however, the problem went away, likely due to the legacy notebook settings.

[sudipdbx]

I had resolved it by upgrading couple of things - 1) in environment, bumped-up from 'Standard V2" to "Standard V4" and 2) at "_common" [or 'setup'] notebook, bumped databricks-sdk from "0.30.0" [default comes with dbc"] to the latest e.g. "0.60.0" ..... now all MLLIBs are working as expected