py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.feature.VectorAssembler(java.lang.String) is not whitelisted.

[DataRabbit]

Hello, i have a problem.

When I try to run the MLlib Assembler (from pyspark.ml.feature import VectorAssembler) I get this error and I don't know what to do anymore. Please help.

[Hubert-Dudek · Accepted solution]

Is it High Concurrency cluster with credential passthrough enabled? In that case, you can use a different cluster mode.

https://docs.azuredatabricks.net/spark/latest/data-sources/azure/adls-passthrough.html

This exception is thrown when you have accessed a method that Azure Databricks has not explicitly marked as safe for Azure Data Lake Storage credential passthrough clusters. In most cases, the technique could allow a user on an Azure Data Lake Storage credential passthrough cluster to access another user’s credential.

[DataRabbit]

Thanks you so much!