topic Re: Not able create table form external spark in Data Engineering

Not able create table form external spark

dixonantony — Thu, 28 Nov 2024 05:22:47 GMT

py4j.protocol.Py4JJavaError: An error occurred while calling o123.sql.
: io.unitycatalog.client.ApiException: generateTemporaryPathCredentials call failed with: 401 - {"error_code":"UNAUTHENTICATED","message":"Request to generate access credential for path 'abfss://ext-location@databrickspocdixon.dfs.core.windows.net/tables/demoTab3' from outside of Databricks Unity Catalog enabled compute environment is denied for security. Please contact Databricks support for integrations with Unity Catalog.","details":[{"@type":"type.googleapis.com/google.rpc.ErrorInfo","reason":"UNITY_CATALOG_EXTERNAL_GENERATE_PATH_CREDENTIALS_DENIED","domain":"unity-catalog.databricks.com","metadata":{"path":"abfss://ext-location@databrickspocdixon.dfs.core.windows.net/tables/demoTab3"}},{"@type":"type.googleapis.com/google.rpc.RequestInfo","request_id":"dafd42ef-1dd1-4225-b41e-ac44be07e6d5","serving_data":""}]}
at io.unitycatalog.client.api.TemporaryCredentialsApi.getApiException(TemporaryCredentialsApi.java:78)
at io.unitycatalog.client.api.TemporaryCredentialsApi.generateTemporaryPathCredentialsWithHttpInfo(TemporaryCredentialsApi.java:192)
at io.unitycatalog.client.api.TemporaryCredentialsApi.generateTemporaryPathCredentials(TemporaryCredentialsApi.java:170)
at io.unitycatalog.spark.UCSingleCatalog.createTable(UCSingleCatalog.scala:106)
at org.apache.spark.sql.execution.datasources.v2.CreateTableExec.run(CreateTableExec.scala:44)
at org.apache.spark.sql.execution.datasources.v2.V2CommandExec.result$lzycompute(V2CommandExec.scala:43)

Re: Not able create table form external spark

NandiniN — Mon, 02 Dec 2024 11:04:29 GMT

Hello @dixonantony ,

Have you followed the steps here - https://docs.databricks.com/en/data-governance/unity-catalog/access-open-api.html#external-data-access

Thanks!

Re: Not able create table form external spark

NandiniN — Mon, 02 Dec 2024 11:05:46 GMT

Also how are you creating the table,

can you please try the below option

df.write.insertInto , or the v2 version df.writeTo(tbl_name).append()

df.write.saveAsTable does not fully support custom catalogs.

Re: Not able create table form external spark

dixonantony — Mon, 02 Dec 2024 11:17:32 GMT

Thanks for the reply, I was following below doc and select table is working, issue is with create table.

spark.sql(f"create table if not exists {catalog}.{databasename}.testTable(id INTEGER, name VARCHAR(10), age INTEGER)").show()

https://community.databricks.com/t5/technical-blog/integrating-apache-spark-with-databricks-unity-catalog-assets/ba-p/97533

Same issue is listed in https://github.com/unitycatalog/unitycatalog/issues/560 but not sure what exactly is the SAFE enabling.

Re: Not able create table form external spark

NandiniN — Tue, 03 Dec 2024 03:21:32 GMT

I see thanks for sharing, for the safe flag - You can work with your Databricks Point of Contact to get help on enabling the SAFE flag for the workspace.

Re: Not able create table form external spark

NandiniN — Tue, 03 Dec 2024 03:24:57 GMT

Can you also try this validate on the privileges please

Grant CREATE EXTERNAL TABLE on the external location to create a delta table. on UNITY_CATALOG_EXTERNAL_GENERATE_PATH_CREDENTIALS_DENIED because that privilege is blocked behind.

Re: Not able create table form external spark

dixonantony — Tue, 03 Dec 2024 03:40:30 GMT

I already tried CREATE EXTERNAL TABLE , but not working.

Re: Not able create table form external spark

NandiniN — Wed, 08 Jan 2025 23:33:53 GMT

There also is this blog https://community.databricks.com/t5/technical-blog/integrating-apache-spark-with-databricks-unity-catalog-assets/ba-p/97533 that came up while reviewing your issue. All of this is in public preview.

Re: Not able create table form external spark

NandiniN — Fri, 10 Jan 2025 16:37:21 GMT

You need the generateTemporaryPathCredentials API as you are trying to create external tables