Databricks Community

Iceberg Interoperability (Databricks <> Snowflake) In this post, we’ll explore the two main approaches to integrating UC with other Iceberg environments — Foreign Iceberg tables and Managed Iceberg tables — and walk through when to choose each, how t...

Databricks Auto Loader is a powerful tool for efficiently ingesting files from cloud storage. In this blog, we'll focus on setting up Auto Loader in file notification mode on AWS using service credentials. Why Use File Notification Mode? File notific...

As organizations continue to scale their data infrastructure, efficient resource utilization, cost control, and operational transparency are paramount for success. With the growing adoption of Databricks, monitoring and optimizing compute usage and d...

IntroductionCreate S3 bucket and cross-account instance profileUse Autoloader to create SNS-SQS across accountsManually create SNS-SQS for cross-account AutoloaderTest cross-account Autoloader connection Introduction This is a step-by-step guide to s...

Databricks Unity Catalog (UC) hosts System Tables which contain the operational data of the account. There are tables to track audit logs, billing, lineage, and more.  In this article, we will see how to use the system.billing.usage table to get insi...