External Table From BigQuery

[Kayla]

I'm working on implementing Unity Catalog, and part of that is determining how to handle our BigQuery tables. We need to utilize them to connect to another application, or else we'd stay within regular delta tables on Databricks.
The page https://docs.gcp.databricks.com/external-data/bigquery.html#read-and-write-to-a-bigquery-table gives information on how to connect, but makes it clear that it will not work with Unity Catalog. Is there a way to do this with Unity Catalog?

[Kaniz]

Hi @Kayla, Certainly! Let’s discuss how Unity Catalog can help you manage your data and analytics assets, including BigQuery tables:

What is Unity Catalog?

• Unity Catalog is Databricks’ unified data, analytics, and AI governance solution on the lakehouse. It comprehensively manages structured and unstructured data, machine learning models, notebooks, dashboards, and files across various clouds and platforms.
• With Unity Catalog, data scientists, analysts, and engineers can securely discover, access, and collaborate on trusted data and AI assets, leveraging AI to boost productivity and unlock the full potential of the lakehouse environment.

Lakehouse Federation Capabilities:

• Unity Catalog now offers Lakehouse Federation capabilities, allowing you to:
  • Discover, query, and govern data across various data platforms, including Google BigQuery, without moving or copying the data.
  • Consolidate, map, and query data from platforms such as MySQL, PostgreSQL, Amazon Redshift, Snowflak....
  • This means you can work directly with BigQuery tables from Databricks without compromising governance or data management.

Benefits of Unity Catalog:

• Unified Visibility:
  • Discover and classify structured and unstructured data, ML models, notebooks, dashboards, and arbitrary files on any cloud.
  • Accelerate data and AI initiatives with a single access point for data exploration.
• Single Permission Model:
  • Simplify access management with a unified interface to define access policies on data and AI assets.
  • Securely access data from other computing platforms using open interfaces, with consistent permissions managed in one place.
• AI-Powered Monitoring and Observability:
  • Automate monitoring, diagnose errors and uphold data and ML model quality.
  • Benefit from proactive alerts that detect personally identifiable information (PII) data and track model drift.
• Open Data Sharing:
  • Share data and AI assets across clouds, regions, and platforms with open source Delta Sharing, nativ....

Connecting to BigQuery with Unity Catalog:

• While the official documentation states that direct reading and writing to BigQuery tables won’t work with Unity Catalog, the Lakehouse Federation capabilities now allow you to work with BigQuery data seamlessly.
• Without data movement or replication, you can discover, query, and govern BigQuery tables from within Databricks.

In summary, Unity Catalog provides a unified governance solution that bridges the gap between data platforms, enabling you to work with BigQuery tables alongside other data sources. 

Feel free to explore this powerful feature to enhance your data workflows! 😊.