Error "Integrating Apache Spark with Databricks Unity Catalog Assets via Open APIs" on Azure

[charl-p-botha]

Great blog post: https://community.databricks.com/t5/technical-blog/integrating-apache-spark-with-databricks-unity-ca...

I have attempted to reproduce this with Azure Databricks, and ADLS gen2 as the storage backend.

Although I'm able to interact with unity catalog (successful "use schema" and then "select(current_schema()") and so on, when I try to append rows to a newly created managed table (as in the example above), I get the error below.

It looks like the temporary credential supplied by UC is failing. Any ideas what could be wrong here?

java.nio.file.AccessDeniedException: Operation failed: "This request is not authorized to perform this operation using this permission.", 403, PUT, https://storageaccout-xyz.dfs.core.windows.net/some/dir/__unitycatalog/..."

Any ideas what I'm doing wrong? cc @dkushari 

P.S. it looks like the TM in the title of your blog post is preventing anyone from commenting there.

P.P.S. this forum software is extremely painful. It won't accept the post without a label, but it makes it incredibly difficult to select any labels.

[dkushari]

Hi @charl-p-botha - Thanks for your post. Please see if you have all the right UC permissions to modify the table. Can you run the same command from a DB workspace and see if it passes? Please make sure all of these are satisfied. I will check on the P.S and P.P.S with the team.

[dkushari]

Hi @charl-p-botha - Can you please ensure that you are logged into the Databricks community portal to put the comments for the blog?

[charl-p-botha]

I was definitely logged in. The error message I got each time as I tried to leave a comment was:

"

Correct the highlighted errors and try again.

The message subject contains <TM symbol>, which is not permitted in this community. Please remove this content before sending your post."

(It gave me the error again now here, so I replaced the actual symbol with <TM symbol>. The title of your blog contains the symbol, so all comments will get blocked in the same way.)

[charl-p-botha]

Hi there @dkushari thank you very much for getting back to me!

I have just confirmed that I am able to run exactly the same insert command in a databricks notebook. I am using a PAT token for that user account in my apache spark experiments.

Is there anything else I can try?