I've been working on creating a schema in the Hive Metastore using the following command:
spark.sql('CREATE DATABASE IF NOT EXISTS {database}')The schema or database is successfully created, but I encountered an issue where it's only accessible for me and not for other users. To address this, I attempted to grant select permissions on the schema using the command:
spark.sql('GRANT SELECT ON SCHEMA hive_metastore.{database} TO `{user}`')However, I received the following error:
org.apache.spark.SparkException: Trying to perform a permission action on Hive Metastore /CATALOG/`hive_metastore`/DATABASE/`testing_permissions_acls`, but Table Access Control is not enabled on this cluster.
To resolve this, I added the following configuration setting to SparkConf:
spark.databricks.acl.sqlOnly true
After making this change, I was able to see the schema's permissions tab through the GUI. However, I encountered a new error:
Your administrator has only allowed SQL commands on this cluster. This execution contained at least one disallowed language.
Here is an overview of my cluster configuration:
Cluster Configuration:
- Databricks Runtime: 10.4 LTS ML (includes Apache Spark 3.2.1, Scala 2.12)
- Access Mode: Custom
- IAM role passthrough: Not available (disabled)
Unity Catalog: Not enabled
Spark Configuration:
spark.driver.extraJavaOptions "-Dlog4j2.formatMsgNoLookups=true"
spark.databricks.io.cache.maxMetaDataCache 1g
spark.databricks.io.cache.compression.enabled false
spark.hadoop.fs.s3a.acl.default BucketOwnerFullControl
spark.databricks.libraryIsolation.enabled true
spark.databricks.unityCatalog.userIsolation.python.preview true
spark.hadoop.fs.s3a.canned.acl BucketOwnerFullControl
spark.databricks.io.cache.maxDiskUsage 50g
spark.databricks.acl.sqlOnly true
spark.sql.execution.arrow.enabled true
spark.executor.extraJavaOptions "-Dlog4j2.formatMsgNoLookups=true"
Has anyone encountered a similar issue before or have suggestions on how to resolve it?
Thanks in advance,
Adriana Cavalcanti
