- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-30-2026 03:38 AM
Hi @wanglee ,
1. While your solution is valid, it might turn out to be not the most cost-efficient. While the tables are small (< 50k rows) and assuming they will not grow over time, more cost-effective solution is to define tables (and not materialized views) on the external location. Materialized views are good for materializing heavy aggregations on large amounts of data. However, for <50k rows it can be an overkill.
USE CATALOG example_catalog;
USE CATALOG example_schema_1;
CREATE OR REPLACE TABLE table_1 AS
SELECT * FROM `delta`.`gs://example_bucket/landing/connector_1/table_1`;
2. Databricks won't automatically "scan" a bucket to create a catalog. However, you can automate this using a simple Python notebook. You can use dbutils.fs.ls() to list the folders in your landing zone and dynamically execute the CREATE TABLE statement for each table found. This handles the "50+ tables" problem in a few lines of code.
3. Auto loader would not be better here as the data volume is low (and assuming it wouldn't grow). Simply re-read the whole table at every run.
Hope this helps!