Data profiling monitoring with foreign catalog

[sta_gas]

Hi team,

I’m currently working with Azure Databricks and have created a foreign catalog for my source database in Azure SQL. I can successfully run SELECT statements from Databricks to the Azure SQL database.

However, I would like to set up data profiling monitoring using the Quality tab, but I’m facing limitations in terms of availability and functionality.

The table type is FOREIGN and the catalog type is FOREIGN_CATALOG.

Could you please advise on the best approach or any recommended steps to enable this feature in this catalog? I acknowledge that i can create materialize views or replicate the data into managed tables on another catalog, however I would like not to replicate all the data.

[szymon_dybczak · Accepted solution]

Hi @sta_gas ,

Since data quality monitoring is in beta I'm quite sure they don't support foreign tables as of now (but they forgot to mentioned it in docs).

But more important question if they ever will be supported. For me data quality monitoring applies only to Delta Tables. According to docs description of how it works, we can see that they leverage delta properties to build this functionality. So I guess it won't work for foreign tables (at least there won't be the same feature parity).

"Databricks creates a background job that monitors tables for freshness and completeness. Databricks uses smart scanning to determine when to scan tables.

Freshness refers to how recently a table has been updated. Data quality monitoring analyzes the history of commits to a table and builds a per-table model to predict the time of the next commit. If a commit is unusually late, the table is marked as stale."

[sta_gas]

Hi szymon, 

Thank you for your quick response. I understand that data quality can be more complex. However, I believe that for “Data Profiling” monitoring, this approach could still be valid, as Unity Catalog generates predefined SQL queries to extract statistical and other relevant metrics and this could be done with SQL pushdowns.