Hi @data-wrangler 

This is a common issue when setting up BigQuery as a foreign catalog in Databricks,
especially with cross-project service accounts and limited permissions.
The problem you're experiencing is likely related to how Databricks discovers and catalogs BigQuery metadata.

The most likely culprit is that Databricks needs to query BigQuery's metadata tables to populate the foreign catalog,
but your service account has limited access. Even though you can query specific tables directly,
the catalog discovery process requires broader metadata access.

Solutions to Try:
1. Request Additional IAM Roles
Ask your vendor to grant your service account these additional roles on the BigQuery project:
- roles/bigquery.metadataViewer - This is crucial for catalog discovery
- roles/bigquery.resourceViewer - Helps with project-level resource discovery
These roles allow reading metadata without granting data access beyond what you already have.

2. Specify the Correct Project in Catalog Creation
When creating the foreign catalog, use the BigQuery project ID (where the data lives), not your service account's project.
The connection can use your SA project, but the catalog needs to point to the data project.

3. Try Dataset-Specific Catalog Creation
Instead of cataloging the entire project, try creating a catalog that points to specific datasets you have access to.
Some Databricks versions allow you to specify dataset-level scope.

4. Manual Table Registration
As a workaround, you might be able to manually register specific tables in the catalog using SQL commands like:

CREATE TABLE catalog_name.schema_name.table_name
USING bigquery
OPTIONS (
table 'project.dataset.table'
)