aleksandra_ch
Databricks Employee
Databricks Employee

Hi @wanglee ,

1. While your solution is valid, it might turn out to be not the most cost-efficient. While the tables are small (< 50k rows) and assuming they will not grow over time, more cost-effective solution is to define tables (and not materialized views) on the external location. Materialized views are good for materializing heavy aggregations on large amounts of data. However, for <50k rows it can be an overkill. 

USE CATALOG example_catalog;
USE CATALOG example_schema_1;

CREATE OR REPLACE TABLE table_1 AS
SELECT * FROM `delta`.`gs://example_bucket/landing/connector_1/table_1`;

 2. Databricks won't automatically "scan" a bucket to create a catalog. However, you can automate this using a simple Python notebook. You can use dbutils.fs.ls() to list the folders in your landing zone and dynamically execute the CREATE TABLE statement for each table found. This handles the "50+ tables" problem in a few lines of code.

3. Auto loader would not be better here as the data volume is low (and assuming it wouldn't grow). Simply re-read the whole table at every run.

Hope this helps!

View solution in original post