I've been working on creating a schema in the Hive Metastore using the following command:
spark.sql(f'CREATE DATABASE IF NOT EXISTS {database}')The schema or database is successfully created, but I encountered an issue where it's only accessible for me and not for other users. To address this, I attempted to grant select permissions on the schema using the command:
spark.sql(f'GRANT SELECT ON SCHEMA hive_metastore.{database} TO `{user}`')However, I received the following error:
org.apache.spark.SparkException: Trying to perform a permission action on Hive Metastore /CATALOG/`hive_metastore`/DATABASE/`testing_permissions_acls`,
but Table Access Control is not enabled on this cluster.
To resolve this, I added the following configuration setting to SparkConf:
spark.databricks.acl.sqlOnly true
After making this change, I was able to see the schema's permissions tab through the GUI. However, I encountered a new error:
Your administrator has only allowed SQL commands on this cluster.
This execution contained at least one disallowed language.
Here is an overview of my cluster configuration:
Cluster Configuration:
- Databricks Runtime: 10.4 LTS ML (includes Apache Spark 3.2.1, Scala 2.12)
- Access Mode: Custom
- IAM role passthrough: Disabled (Cant tick the box)
Unity Catalog: Not Enabled
Spark Configuration:
- `spark.driver.extraJavaOptions`: "-Dlog4j2.formatMsgNoLookups=true"
- `spark.databricks.io.cache.maxMetaDataCache`: 1g
- `spark.databricks.io.cache.compression.enabled`: false
- `spark.hadoop.fs.s3a.acl.default`: BucketOwnerFullControl
- `spark.databricks.libraryIsolation.enabled`: true
- `spark.databricks.unityCatalog.userIsolation.python.preview`: true
- `spark.hadoop.fs.s3a.canned.acl`: BucketOwnerFullControl
- `spark.databricks.io.cache.maxDiskUsage`: 50g
- `spark.databricks.acl.sqlOnly`: true
- `spark.sql.execution.arrow.enabled`: true
- `spark.executor.extraJavaOptions`: "-Dlog4j2.formatMsgNoLookups=true"
Has anyone encountered a similar issue before or have suggestions on how to resolve it?
Thanks in advance,
Adriana Cavalcanti
