Databricks Governance and DWH Announcements — July 2024

[youssefmrini]

Governance & Delta Sharing

New connection support for Lakehouse federation

You can run federated queries on data managed by Salesforce Data Cloud

You can use SSO authentication to connect to SQL Server

Query History and Node timeline are now available in System Tables

• Query history show sql queries run by SQL warehouses. It’s powerful to track your queries (execution status, error, data management etc). Learn more

• Node timeline captures node-level resource utilization data at minute granularity. Each record contains data for a given minute of time per instance. Learn more

Partition metadata logging for UC external tables

For DBR13.3 LTS or above, you can enable partition metadata logging, which is a partition discovery strategy for external tables registered to Unity Catalog. The behavior is consistent with the partition discovery strategy used in Hive metastore and only impacts Unity Catalog external tables that have partitions and use Parquet, ORC, CSV, or JSON.

Databricks recommends enabling the new behavior for improved read speeds and query performance for these tables.Learn more

What’s new in Delta Sharing ?

Delta Sharing lets you share tables where liquid clustering is enabled as well as the objects metadata including comments and primary constraints and AI Models.

Liquid Clustering Sharing Support AI Models Sharing

Compute & Data Engineering

Databricks Lakeflow Connect is available

LakeFlow Connect offers native connectors that enable you to ingest data from databases and enterprise applications and load it into Databricks. LakeFlow Connect leverages efficient incremental reads and writes to make data ingestion faster, scalable, and more cost-efficient, while your data remains fresh for downstream consumption.

Salesforce Sales Cloud, Microsoft Azure SQL Database, Amazon RDS for SQL Server, and Workday Reports-as-a-Service (RaaS) are currently supported.

Blog Announcement, Keynote presentation

Databricks Serverless Compute for Workflows and Notebooks are GA

Configuring and managing compute such as Spark clusters has long been a challenge for data engineers and data scientists. Time spent on configuring and managing compute is time not spent providing value to the business.

Databricks Connect for Python now supports Serverless Compute

 

Data Warehousing

Predictive Optimisation, which can improve your query performance by 2x through intelligent optimization of data layouts, is now in GA. Learn more

Cost management dashboards are now in Public preview, making it easy to import a dashboard to monitor costs on a workspace or account level. Learn more

In a nutshell

• In Databricks Runtime 15.4 LTS and above, Scala is generally available on shared access mode Unity Catalog-enabled compute, including support for scalar user-defined functions (UDFs)

• Account SCIM v2.1. Learn more

• End of life for Databricks managed passwords. Learn more

• Databricks provides an open source software (OSS) JDBC driver that enables you to connect tools such as DataGrip, DBeaver, and SQL Workbench/J to Databricks through Java Database Connectivity (JDBC), an industry-standard specification for accessing database management systems.

• Library installation on clusters now has a timeout of 2 hours.