cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Importing data from S3 to Azure DataBricks Cluster with Unity Catalog in Shared Mode

sha
New Contributor

Environment details:

DataBricks on Azure, 13.3 LTS, Unity Catalog, Shared Cluster mode.

Currently in the environment I'm in, we run imports from S3 with code like:

spark.read.option('inferSchema', 'true').json(s3_path). 
 
When running on a cluster in Shared Mode with Unity Catalog enabled, I get this error:
"Import for <table> failed with error: An error occurred while calling o453.json. : org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission SELECT on any file."
 
There's a proposed workaround , but this isn't possible since I don't have admin access and the admins don't want to bypass all the security controls provided by Unity Catalog. Running the code in Single User mode works with no issues, but having a bunch of Single User mode clusters to support my team isn't a feasible solution.
 
Basic question is: what mechanisms can be used to import S3 data into a Unity Catalog enabled Shared Cluster environment, if any, without resorting to being a cluster admin? 
1 REPLY 1

BR_DatabricksAI
Contributor

Hello Sha, 

We usually get such error while working with shared cluster mode assuming this your dev environment just to avoid errors, please use different clusters. 

However as a alternative solution in case if would like to keep the shared cluster then you create a group and assign multiple users in the groups and then request admin to grant the necessary select privileges on specific catalog, schema, view and tables. 

Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!