topic Re: py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.feature.VectorAssembler(java.lang.String) is not whitelisted. in Data Engineering

py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.feature.VectorAssembler(java.lang.String) is not whitelisted.

DataRabbit — Wed, 12 Oct 2022 09:05:40 GMT

Hello, i have a problem.

When I try to run the MLlib Assembler (from pyspark.ml.feature import VectorAssembler) I get this error and I don't know what to do anymore. Please help.

Re: py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.feature.VectorAssembler(java.lang.String) is not whitelisted.

Hubert-Dudek — Sun, 16 Oct 2022 18:48:36 GMT

Is it High Concurrency cluster with credential passthrough enabled? In that case, you can use a different cluster mode.

https://docs.azuredatabricks.net/spark/latest/data-sources/azure/adls-passthrough.html

This exception is thrown when you have accessed a method that Azure Databricks has not explicitly marked as safe for Azure Data Lake Storage credential passthrough clusters. In most cases, the technique could allow a user on an Azure Data Lake Storage credential passthrough cluster to access another user’s credential.

Re: py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.feature.VectorAssembler(java.lang.String) is not whitelisted.

DataRabbit — Tue, 25 Oct 2022 20:04:00 GMT

Thanks you so much!

Re: py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.feature.VectorAssemb

DBXC — Thu, 10 Oct 2024 15:50:04 GMT

How could we whitelist this error below with DBR 13.3 and above?

Py4JError: An error occurred while calling None.org.apache.spark.ml.recommendation.ALS. Trace: py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.recommendation.ALS(java.lang.String) is not whitelisted. at py4j.security.WhitelistingPy4JSecurityManager.checkConstructor(WhitelistingPy4JSecurityManager.java:451) at py4j.Gateway.invoke(Gateway.java:256) at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80) at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69) at py4j.ClientServerConnection.waitForCommands(ClientServerConnection.java:199) at py4j.ClientServerConnection.run(ClientServerConnection.java:119) at java.lang.Thread.run(Thread.java:750)

Re: py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.feature.VectorAssemb

SudiptaDe — Fri, 11 Jul 2025 01:17:27 GMT

I am facing the same error with Databricks Serverless (Free Edition) .... please fix it

Re: py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.feature.VectorAssemb

VenuG — Tue, 18 Nov 2025 19:26:03 GMT

Do you plan to support this in Serverless Free Edition? Migration from Community Edition to Serveless has been fraught with these limitations.

Re: py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.feature.VectorAssemb

rabbit-task — Sun, 22 Feb 2026 20:44:47 GMT

I encountered the same issue after switching from Databricks Community Edition to Databricks Free Edition. After I detach and reattach the serverless compute, however, the problem went away, likely due to the legacy notebook settings.

Re: py4j.security.Py4JSecurityException: Constructor public org.apache.spark.ml.feature.VectorAssemb

sudipdbx — Mon, 23 Feb 2026 05:19:37 GMT

I had resolved it by upgrading couple of things - 1) in environment, bumped-up from 'Standard V2" to "Standard V4" and 2) at "_common" [or 'setup'] notebook, bumped databricks-sdk from "0.30.0" [default comes with dbc"] to the latest e.g. "0.60.0" ..... now all MLLIBs are working as expected