topic Unity Catalog + Streaming error: method public X is not whitelisted on class DataStreamReader in Data Governance

Unity Catalog + Streaming error: method public X is not whitelisted on class DataStreamReader

data_boy_2022 — Sat, 10 Sep 2022 01:16:41 GMT

I want to import data using the autoloader from a S3 bucket into a table which is managed inside a Unity Catalog.

Right now, I run the code on an interactive cluster inside a notebook. In the future the code should run in a job cluster.

The error I get is the following:

py4j.security.Py4JSecurityException: Method public org.apache.spark.sql.streaming.DataStreamReader org.apache.spark.sql.streaming.DataStreamReader.format(java.lang.String) is not whitelisted on class class org.apache.spark.sql.streaming.DataStreamReader

What have I tried so far:

Enabling credential passthrough on the cluster:

=> Doesn't work since Unity Catalog can't be used with this option

I also tried setup an external location as described here:

https://docs.databricks.com/data-governance/unity-catalog/manage-external-locations-and-credentials.html#create-external-location&language-Python

On top, I have found this article but the solution is not actionable for me:

https://kb.databricks.com/en_US/streaming/readstream-is-not-whitelisted

Can anybody help?

Re: Unity Catalog + Streaming error: method public X is not whitelisted on class DataStreamReader

Tian — Mon, 19 Sep 2022 19:56:15 GMT

Hi!

Databricks recently released the documentation on using Unity Catalog with Structured Streaming: https://docs.databricks.com/structured-streaming/unity-catalog.html

Per document requirement, for both interactive notebooks and scheduled jobs, you must use single user clusters for Structured Streaming on Unity Catalog. Python and Scala are supported. Could you verify if the cluster access model is single user?

Re: Unity Catalog + Streaming error: method public X is not whitelisted on class DataStreamReader

data_boy_2022 — Tue, 20 Sep 2022 06:10:49 GMT

Works with single user mode. Thank you!