cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Importing data from S3 to Azure DataBricks Cluster with Unity Catalog in Shared Mode

sha
New Contributor

Environment details:

DataBricks on Azure, 13.3 LTS, Unity Catalog, Shared Cluster mode.

Currently in the environment I'm in, we run imports from S3 with code like:

spark.read.option('inferSchema', 'true').json(s3_path). 
 
When running on a cluster in Shared Mode with Unity Catalog enabled, I get this error:
"Import for <table> failed with error: An error occurred while calling o453.json. : org.apache.spark.SparkSecurityException: [INSUFFICIENT_PERMISSIONS] Insufficient privileges: User does not have permission SELECT on any file."
 
There's a proposed workaround , but this isn't possible since I don't have admin access and the admins don't want to bypass all the security controls provided by Unity Catalog. Running the code in Single User mode works with no issues, but having a bunch of Single User mode clusters to support my team isn't a feasible solution.
 
Basic question is: what mechanisms can be used to import S3 data into a Unity Catalog enabled Shared Cluster environment, if any, without resorting to being a cluster admin? 
1 REPLY 1

BR_DatabricksAI
Contributor

Hello Sha, 

We usually get such error while working with shared cluster mode assuming this your dev environment just to avoid errors, please use different clusters. 

However as a alternative solution in case if would like to keep the shared cluster then you create a group and assign multiple users in the groups and then request admin to grant the necessary select privileges on specific catalog, schema, view and tables. 

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.