Error "Integrating Apache Spark with Databricks Unity Catalog Assets via Open APIs" on Azure

[charl-p-botha]

Great blog post: https://community.databricks.com/t5/technical-blog/integrating-apache-spark-with-databricks-unity-ca...

I have attempted to reproduce this with Azure Databricks, and ADLS gen2 as the storage backend.

Although I'm able to interact with unity catalog (successful "use schema" and then "select(current_schema()") and so on, when I try to append rows to a newly created managed table (as in the example above), I get the error below.

It looks like the temporary credential supplied by UC is failing. Any ideas what could be wrong here?

java.nio.file.AccessDeniedException: Operation failed: "This request is not authorized to perform this operation using this permission.", 403, PUT, https://storageaccout-xyz.dfs.core.windows.net/some/dir/__unitycatalog/..."

Any ideas what I'm doing wrong? cc @dkushari 

P.S. it looks like the TM in the title of your blog post is preventing anyone from commenting there.

P.P.S. this forum software is extremely painful. It won't accept the post without a label, but it makes it incredibly difficult to select any labels.

[dkushari]

Hi @charl-p-botha - Thanks for your post. Please see if you have all the right UC permissions to modify the table. Can you run the same command from a DB workspace and see if it passes? Please make sure all of these are satisfied. I will check on the P.S and P.P.S with the team.

[dkushari]

Hi @charl-p-botha - Can you please ensure that you are logged into the Databricks community portal to put the comments for the blog?

[charl-p-botha]

I was definitely logged in. The error message I got each time as I tried to leave a comment was:

"

Correct the highlighted errors and try again.

The message subject contains <TM symbol>, which is not permitted in this community. Please remove this content before sending your post."

(It gave me the error again now here, so I replaced the actual symbol with <TM symbol>. The title of your blog contains the symbol, so all comments will get blocked in the same way.)

[charl-p-botha]

Hi there @dkushari thank you very much for getting back to me!

I have just confirmed that I am able to run exactly the same insert command in a databricks notebook. I am using a PAT token for that user account in my apache spark experiments.

Is there anything else I can try?

[charl-p-botha]

Aaah, the link you shared has the following:

"

• Only tables are supported during the public preview.
  • External tables support read and write.
  • Managed tables can only be read."

... and I was trying to append to a managed table.

That is a plausible explanation for the error I saw. However, in your demo, you were able to write to a managed table from the terminal: "You can insert data into the managed table from the local terminal as well as the Databricks workspace. First, insert and select some data from the local terminal:" 

-- perhaps a good idea to add a note to the post that that is not yet possible in the current public preview?

In my case, I will then have to try the external table option, as for my use case I wish to be able to write to databricks unity catalog from my local spark.