Unable to use dbutils in Premium

[Jain]

I am unable to use dbutils commands and mkdir, etc also does not work after upgrading my Databricks Workspace from Standard tier to Premium tier.

It throws the following error:

py4j.security.Py4JSecurityException: Constructor public com.databricks.backend.daemon.dbutils.FSUtilsParallel(org.apache.spark.SparkContext) is not whitelisted

Any suggestions please ?

Please let me know if you need more information

[daniel_sahal]

@Abhishek Jain​ 

It looks like someone had the same issue before.

https://community.databricks.com/s/question/0D53f00001OFuWLCA1/can-you-help-with-this-error-please-i...

[Jain]

Hi @Daniel Sahal​ , Thank you for a prompt response.

I did have a look at the above.

However, the best commented response throws me this error.

AnalysisException: Cannot modify the value of a static config: spark.databricks.pyspark.enablePy4JSecurity

[Anonymous]

Hi @Abhishek Jain​ 

Hope all is well! Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help. 

We'd love to hear from you.

Thanks!

[Jain]

Hi @Vidula Khanna​ ,

I observed that the Access Mode options available in Databricks by default have different privileges.

When I use Shared Access Mode then the external storage which is mounted is not available and hence dbutils, mkdir, etc commond does not work. However, Table Level Access RBAC works in this.

When I use No Isolation Shared Access Mode then the external storage which is mounted is available and I can use the cluster as required. However, Table Access RBAC does not work in this.

In order to have both available I guess I will have to make a Custom Policy.

For now, I have created 2 clusters with different access modes as a workaround.

I couldn't find any Databricks official documentation which speaks about this case ​hence had to Hit & Trial to figure out.

Let me know if I might have missed anything.