Azure Network Settings in regards to Databricks Table Monitoring

AsgerLarsen — Thu, 25 Jul 2024 09:17:39 GMT

I have set up my Unity Catalog on an Azure Data Lake which uses the companies virtual network to allow access.
I have all privileges on my account, so I am able to create, alter or delete catalogs, schemas and tables. I can do these things either using the Databricks UI or simple by using SQL statements from a notebook.

When it comes to the creation of monitoring metric tables, this is not the case with my current network settings.
I get an error when trying to setup the monitoring on tables, saying I don't have access to do these things, and the metrics tables are not created.

I have gone through the proposed checklist in this post, metric-tables-not-created-automatically., and it looks like everything is setup correctly.

If I then switch my Azure Data Lake network settings from virtual network access to enabled access from all networks, then I am able to create the metrics tables without any problems.

Maybe the difference is that my general purpose cluster have access to do everything, but the warehouse cluster that is required when creating the metric tables do not? -I imagine that the general purpose cluster is able to use the companies virtual network and the warehouse cluster is not.

Could this be what is causing the problem?

Should I reach out to Microsoft regarding this network issue instead?

Re: Azure Network Settings in regards to Databricks Table Monitoring

AsgerLarsen — Wed, 31 Jul 2024 06:33:47 GMT

Hi Kaniz,

Thanks for the response and for identifying the problem.

I would like some steps on how to adjust the network settings, as everything I have tried so far, hasn't seemed to work.

Re: Azure Network Settings in regards to Databricks Table Monitoring

AsgerLarsen — Wed, 31 Jul 2024 10:15:05 GMT

I've found a similar issue here on the databricks forum:
https://community.databricks.com/t5/data-engineering/unable-to-read-data-from-adls-using-databricks-serverless-sql/td-p/63684
I'll try what you were suggesting in that post.

topic Re: Azure Network Settings in regards to Databricks Table Monitoring in Administration & Architecture

Azure Network Settings in regards to Databricks Table Monitoring

Re: Azure Network Settings in regards to Databricks Table Monitoring

Re: Azure Network Settings in regards to Databricks Table Monitoring