topic Databricks job fails while creating table. in Data Engineering

Databricks job fails while creating table.

RajeshRK — Wed, 29 Jun 2022 14:10:18 GMT

Hi Team,

The Databricks job fails with the below error while creating EXTERNAL table.

com.simba.spark.jdbc41.internal.apache.http.wire - Error running query: MetaException(message:Got exception: org.apache.hadoop.fs.azure.AzureException com.microsoft.azure.storage.StorageException: Public access is not permitted on this storage account.)

MetaException(message:Got exception: org.apache.hadoop.fs.azure.AzureException com.microsoft.azure.storage.StorageException: Public access is not permitted on this storage account.), Query: CREATE EXTERNAL TABLE

Can you please suggest if there is any permission missing in WASB storage?

Regards,

Rajesh.

Re: Databricks job fails while creating table.

Vivian_Wilfred — Wed, 29 Jun 2022 14:37:08 GMT

Hi @Rajesh Kannan R , Can you verify the networking configuration on your storage account? Check the Firewall rule to see if public access is enabled for all networks. Enable all network and see if it resolves your issue.

If you have restricted public access then the workspace has to be NPIP and you must have all the required workspace traffic to be whitelisted.

https://docs.microsoft.com/en-us/azure/databricks/security/secure-cluster-connectivity

Re: Databricks job fails while creating table.

RajeshRK — Wed, 29 Jun 2022 15:46:17 GMT

@Vivian Wilfred

Hi Vivian,

Thank you for your quick update. Let me check the settings and update you.

Re: Databricks job fails while creating table.

RajeshRK — Thu, 30 Jun 2022 04:05:06 GMT

@Vivian Wilfred

Hi Vivian,

As I checked the network settings for the storage account and it is set to "Enabled from all networks".

Regards,

Rajesh.

Re: Databricks job fails while creating table.

Vivian_Wilfred — Thu, 30 Jun 2022 20:43:15 GMT

@Rajesh Kannan R , try enabling the Blob public access to see if it resolves the issue. Look for this setting in Overview > Properties > Data lake Storage >

Re: Databricks job fails while creating table.

RajeshRK — Fri, 01 Jul 2022 05:15:21 GMT

@Vivian Wilfred

Hi Vivian,

It is working when Blob public access is enabled, but as security concerns, our customers don't want to enable Blob public access. Is there any other way we can publish to databricks table without enabling blob public access?

Re: Databricks job fails while creating table.

RajeshRK — Thu, 07 Jul 2022 15:26:20 GMT

@Vivian Wilfred

Hi Vivian,

Can you please let me know is there any other way to create a table without enabling blob public access?

Regards,

Rajesh.

Re: Databricks job fails while creating table.

Vivian_Wilfred — Thu, 07 Jul 2022 16:51:55 GMT

Hi @Rajesh Kannan R , Allow Blob Public Access means that a storage account allows public blob/container access levels on the containers. If this is disabled then it means all containers must be private. When containers are private then you must authenticate with key, sp, etc. Please validate the authentication for these containers.

https://azure.microsoft.com/en-in/updates/choose-to-allow-or-disallow-blob-public-access-on-azure-storage-accounts/

Re: Databricks job fails while creating table.

RajeshRK — Mon, 11 Jul 2022 15:54:52 GMT

@Vivian Wilfred

Hi Vivian,

Thank you for the update. I will check and get back to you.

Regards,

Rajesh.

Re: Databricks job fails while creating table.

RajeshRK — Mon, 22 Aug 2022 11:18:46 GMT

@Arjun Kaimaparambil Rajan

Hi Arjun,

We use sasToken to authenticate blob storage.

Regards

Rajesh.

Re: Databricks job fails while creating table.

Vartika — Tue, 30 Aug 2022 15:06:59 GMT

Hey there @Rajesh Kannan R

Hope all is well!

Just wanted to check in if you were able to resolve your issue and would you be happy to share the solution or mark an answer as best? Else please let us know if you need more help.

We'd love to hear from you.

Thanks!