topic Re: Cannot CREATE TABLE with 'No Isolation Shared' cluster in Data Engineering

Cannot CREATE TABLE with 'No Isolation Shared' cluster

lawrence009 — Fri, 09 Dec 2022 04:39:44 GMT

Recently I ran into a number issues running with our notebooks in Interactive Mode. For example, we can't create (delta) table. The command would run and then idle for no apparent exception. The path is created on AWS S3 but delta log is never created. The cell running the command simply idles.

After much trial and error, it turns out 'No Isolate Shared' was the issue. It seems to make sense in retrospect this would be the issue. Has anyone else run into this problem?

Re: Cannot CREATE TABLE with 'No Isolation Shared' cluster

Harun — Fri, 09 Dec 2022 09:21:10 GMT

Hi @Lawrence Chen if you are an account admin and if you enabled the "Admin Protection" feature means you will have this limitation. To overcome this you can create a non-admin user with "no isolation shared cluster" or you can create other cluster mode with admin access.

Take a look at this document to know more about the "Admin Protection" feature and its benefits and Limitations

https://docs.databricks.com/administration-guide/account-settings/no-isolation-shared.html

Re: Cannot CREATE TABLE with 'No Isolation Shared' cluster

youssefmrini — Fri, 09 Dec 2022 15:54:31 GMT

The Admin can disable the possibility to use the no Isolate Shared cluster. I recommend you to switch to Single user where UC is activated. Don't worry you won't need to change your code. If you encounter this kind of issues, make sure to open a ticket

Re: Cannot CREATE TABLE with 'No Isolation Shared' cluster

lawrence009 — Mon, 12 Dec 2022 23:02:09 GMT

I have confirmed my issue is indeed caused by having 'Admin Protection' enabled. However, I haven't enabled UC in my workspace.

It makes sense, however, to avoid No Isolation Shared and Switched to Single User mode.

Related issue, however, is that under 'Shared Access' mode cluster with Instance Profile attached, I am getting the following error

java.util.concurrent.ExecutionException: shaded.databricks.org.apache.hadoop.fs.s3a.AWSClientIOException: getFileStatus on s3a://.../_delta_log:

com.amazonaws.AmazonClientException: No AWS Credentials provided by AwsCredentialContextTokenProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [com.databricks.backend.daemon.driver.aws.AwsLocalCredentialContextTokenProvider@124d1895: No role specified and no roles available., com.databricks.backend.daemon.driver.aws.ProxiedIAMCredentialProvider@35e6f19a: User does not have any IAM roles]: No AWS Credentials provided by AwsCredentialContextTokenProvider : com.amazonaws.SdkClientException: Unable to load AWS credentials from any provider in the chain: [com.databricks.backend.daemon.driver.aws.AwsLocalCredentialContextTokenProvider@124d1895: No role specified and no roles available., com.databricks.backend.daemon.driver.aws.ProxiedIAMCredentialProvider@35e6f19a: User does not have any IAM roles]

Re: Cannot CREATE TABLE with 'No Isolation Shared' cluster

lawrence009 — Mon, 12 Dec 2022 23:02:54 GMT

Thx @Harun Raseed Basheer . Please also see my reply to @Youssef Mrini