topic Re: System tables in Azure with private link in Administration & Architecture

System tables in Azure with private link

Matt4 — Thu, 18 Jan 2024 11:46:17 GMT

Hi,

Is there any restriction on what is able to be enabled with System tables when using Azure Databricks with a private link connection? And is there a way around the restriction (that is secure)?

In a public network I am able to enable the following in Azure (North Europe)

{"schemas":[{"schema":"storage","state":"ENABLE_COMPLETED"},
{"schema":"operational_data","state":"UNAVAILABLE"},
{"schema":"access","state":"ENABLE_COMPLETED"},
{"schema":"billing","state":"ENABLE_COMPLETED"},
{"schema":"compute","state":"ENABLE_COMPLETED"},
{"schema":"marketplace","state":"ENABLE_COMPLETED"},
{"schema":"lineage","state":"ENABLE_COMPLETED"},
{"schema":"information_schema","state":"ENABLE_COMPLETED"}]}*

In a private network in UK South, I get the following options:

{"schemas":[{"schema":"operational_data","state":"UNAVAILABLE"},{"schema":"access","state":"ENABLE_COMPLETED"},
{"schema":"billing","state":"ENABLE_COMPLETED"},
{"schema":"compute","state":"ENABLE_INITIALIZED"},
{"schema":"marketplace","state":"ENABLE_COMPLETED"},{"schema":"information_schema","state":"ENABLE_COMPLETED"}]}*

The compute one specifically says the following when trying to enable it. Anyone know why it is unavailable?

{"error_code":"INTERNAL_ERROR","message":"System schema \"compute\" is not currently available for enablement","details":[{"@type":"type.googleapis.com/google.rpc.RequestInfo","request_id":"624a5858-4969-4d1f-9704-c4213818d69e","serving_data":""}]}*

Thanks in advance

Re: System tables in Azure with private link

Debayan — Thu, 18 Jan 2024 19:41:56 GMT

Hi, There are few restrictions, such as To enable system tables, you might need to grant network access to the system tables Blob storage endpoint. You can refer to https://learn.microsoft.com/en-us/azure/databricks/administration-guide/system-tables/ and https://learn.microsoft.com/en-us/azure/databricks/security/network/classic/private-link-standard

Re: System tables in Azure with private link

Matt4 — Tue, 23 Jan 2024 08:10:35 GMT

Turns out it was an issue with Microsoft somewhere, they've now fixed it and all the system schemas have been enabled that I can enable

"schemas":[{"schema":"operational_data","state":"UNAVAILABLE"},{"schema":"access","state":"ENABLE_COMPLETED"},{"schema":"billing","state":"ENABLE_COMPLETED"},{"schema":"compute","state":"ENABLE_COMPLETED"},{"schema":"marketplace","state":"ENABLE_COMPLETED"},{"schema":"information_schema","state":"ENABLE_COMPLETED"}]}*