Metric Tables not created automatically

[yatharth]

I was trying the new lakehouse monitoring feature for one of my unity tables, and when I create a monitoring dashboard for my table, the 2 metric tables({output_schema}.{table_name}_profile_metrics and {output_schema}.{table_name}_drift_metrics) are not getting created automatically in the specified path. Am i doing something wrong?

Steps followed:

1.Made sure I was the owner of the table
2.Clicked on the quality tab and get started
3.Chose snapshot profile as Profile type
4.Schedule: Refresh manually
5.Metric tables schema name: catalog_name.output_schema_name, output_schema_name was different from where my table was, but catalog was identical
6.Create dashboard
After this i noticed the 2 metric tables were not created automatically and the dashboard could not be created as these metric tables were missing from the specified path

[Kaniz · Accepted solution]

Hi @yatharth, 

1. Ownership and Permissions:

• You mentioned that you ensured you were the owner of the table. Great! Double-check that you have the necessary permissions to create tables in the specified schema.
• Confirm that you have the appropriate permissions to create tables in the target schema where the metric tables should reside.

1. Schema Names:

   • You mentioned that the output_schema_name for the metric tables was different from where your table is. Ensure that the specified schema exists and is accessible.
   • Verify that the schema name is correctly spelled and matches the actual schema in your Databricks environment.

2. Catalog Name:

   • You mentioned that the catalog was identical. Make sure that the catalog name is accurate and points to the correct data catalog (e.g., Delta Lake, AWS Glue, etc.).
   • If you’re using a different catalog, ensure that it supports the creation of metric tables.

3. Metric Table Creation:

   • When you create the dashboard, Databricks should automatically create the metric tables if they don’t exist.
   • Check the Databricks UI logs or the job history to see if there were any errors during the dashboard creation process.
   • Manually verify whether the metric tables were created in the specified schema.

4. Custom Metrics:

   • Databricks Lakehouse Monitoring allows custom metrics beyond the automatically calculated ones. You can create aggregate, derived, and drift metrics¹.
   • If you’re using custom metrics, ensure that their definitions are correctly specified.

5. API Reference:

   • If you’re comfortable with APIs, you can also create monitors using the API².

Remember that Databricks Lakehouse Monitoring is still in public preview, so there might be occasional quirks.

Keep an eye on the Databricks documentation for any updates or additional troubleshooting steps. 🚀

[Cas]

Any success/workaround on this topic? I have the same issue

[yatharth]

For me getting the admin access for the unity catalog solved it, as only admins can create tables, hence if tables are not getting created automatically, I'll suggest to check if you have permission to create any sort of table manually using the UI or code