topic Re: Unity Catalog - Writing to PNG Files to Cluster and then using dbutils.fs.cp to send to Azure AD in Data Engineering

Unity Catalog - Writing to PNG Files to Cluster and then using dbutils.fs.cp to send to Azure ADLS2

aicd_de — Wed, 20 Sep 2023 00:01:46 GMT

Hi All

Looking to get some help. We are on Unity Catalog in Azure. We have a requirement to use Python to write out PNG files (several) via Matplotlib and then drop those into an ADLS2 Bucket. With Unity Catalog, we can easily use dbutils.fs.cp or fs.put to do this. However, the PNGs need to be written to the Cluster first before we use Copy to move them over to an ADLS2 Bucket.

The issue: dbutils cannot access all locations on the Cluster and the folders it can access we get ERROR 13 Access Denied when trying to write PNGs to those spots. So I am not sure where i can drop the files to do the copy with. Here is the code snippet:

<CODE HERE TO GENERATE CHART>... FOLLOWED BY..

img_name = f'{product_level_1}-{product_level_2}-{product_level_3}-{value_type_1}-{value_type_2}.png'

plt.savefig('/databricks-datasets/'+img_name, bbox_inches='tight', format='png')

dbutils.fs.cp('/databricks-datasets/'+img_name, storage_url+img_name)

print(img_name)

So, the plt.save if I just go img_name, it drops to default workspace location but then dbutils cannot locate it. Then when I try to select folders dbutils can have it doesn't work with permission issues.

Re: Unity Catalog - Writing to PNG Files to Cluster and then using dbutils.fs.cp to send to Azure AD

aicd_de — Thu, 21 Sep 2023 00:20:15 GMT

I get this error writing to that location:

"java.lang.SecurityException: Cannot use com.databricks.backend.daemon.driver.WorkspaceLocalFileSystem - local filesystem access is forbidden"

Re: Unity Catalog - Writing to PNG Files to Cluster and then using dbutils.fs.cp to send to Azure AD

aicd_de — Fri, 22 Sep 2023 02:51:32 GMT

Hmm I read something different - someone else had this error because they used a shared cluster - apparently it does not happen on a single user cluster. All those settings are already done and I am a fully admin.