Databricks Community

KIRKQUINBAR · 3 weeks ago

We started migrating databases from hive_metastore into unity catalog back in October 2024 and ive noticed that periodically the Catalog UI will not show columns or a data preview for some tables, but not all of them that were migrated. After some digging i realized that some tables, the columns are not being populated into the system information_schema columns table which may explain the UI issue.

I was able to recreate the problem using some simple queries. This happens on an existing databricks instance we have had running for years where i added unity catalog on top of. If I create a brand new databricks instance, the columns being populated in system information_schema does not occur. Any one experience this issue. The table itself does show up in information_schema tables table.

select * from system.information_schema.tables where table_catalog='development' and table_schema='default' and table_name='test_base';
select * from system.information_schema.columns where table_catalog='development' and table_schema='default' and table_name='test_base';

select * from system.information_schema.tables where table_catalog='development' and table_schema='default' and table_name='test_copy';
select * from system.information_schema.columns where table_catalog='development' and table_schema='default' and table_name='test_copy';

-- run these together and table columns WILL NOT populate
drop table if exists development.default.test_base;
create table development.default.test_base (
    id int,
    display_name string
);

-- run this without dropping table and table columns WILL populate
-- if table is dropped first then table columns WILL NOT populate
create or replace table development.default.test_base (
    id int,
    display_name string
);

-- run these together and table columns WILL NOT populate
drop table if exists development.default.test_copy;
CREATE TABLE development.default.test_copy
   DEEP CLONE development.default.test_base;

-- run this without dropping table and table columns WILL populate
-- if table is dropped first then table columns WILL NOT populate
CREATE OR REPLACE TABLE development.default.test_copy
   DEEP CLONE development.default.test_base;

This seems like a bug or setting that didnt get set when i converted this existing databricks instance to use unity catalog.