Failed to run the job on databricks version LTS 9.x and 10.x(AWS)

[Paval]

Hi Team,

When we tried to change the databricks version from 7.3 to 9.x or 10.x we are getting below error. 

Caused by: java.lang.RuntimeException: MetaException(message:Unable to verify existence of default database: com.amazonaws.services.glue.model.AccessDeniedException: User: arn:aws:sts::9999999999999:assumed-role/DaspDatabricksEc2DaspTechDev/i-0b910000000419 is not authorized to perform: glue:GetDatabase on resource: arn:aws:glue:eu-west-1:99999999999:catalog because no identity-based policy allows the glue:GetDatabase action (Service: AWSGlue; Status Code: 400; Error Code: AccessDeniedException; Request ID: 04375935-b50f-4c96e229d78c; Proxy: null))
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:522)
at org.apache.spark.sql.hive.client.HiveClientImpl.newState(HiveClientImpl.scala:177)
at org.apache.spark.sql.hive.client.HiveClientImpl.<init>(HiveClientImpl.scala:131)
... 63 more
Caused by: MetaException(message:Unable to verify existence of default database: com.amazonaws.services.glue.model.AccessDeniedException: User: arn:aws:sts::99999999999:assumed-role/DaspDatabricksEc2DaspTechDev/i-0b910000000419 is not authorized to perform: glue:GetDatabase on resource: arn:aws:glue:eu-west-1:9999999999999:catalog because no identity-based policy allows the glue:GetDatabase action (Service: AWSGlue; Status Code: 400; Error Code: AccessDeniedException; Request ID: 04375935-b50f-4c96e229d78c; Proxy: null))
at com.amazonaws.glue.catalog.metastore.AWSCatalogMetastoreClient.doesDefaultDBExist(AWSCatalogMetastoreClient.java:236)
at com.amazonaws.glue.catalog.metastore.AWSCatalogMetastoreClient.<init>(AWSCatalogMetastoreClient.java:141)
at com.amazonaws.glue.catalog.metastore.AWSGlueDataCatalogHiveClientFactory.createMetaStoreClient(AWSGlueDataCatalogHiveClientFactory.java:16)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3028)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3063)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:503)

is there any difference between 7.3 and 9.x or 10.x

[Kaniz]

Hi @Paval, Could you please verify that the cluster uses the IAM policy attached to the IAM role to access the AWS Glue Catalog? Ensure that it includes the glue:GetDatabase action.

If the policy is correct, you must ensure the cluster has the necessary permissions to access the AWS Glue Catalog by configuring the required permissions. You could also check if the policy for accessing the AWS Glue Catalog has changed between Databricks versions.

The error message you are receiving is related to an AWS Glue access issue and is not associated with any differences between Databricks versions.