Planning a Migration to Unity Catalog
Authors: Dhaval Bagadia, Ziyuan Qin Are you using Hive Metastore on Databricks or an external Hive Metastore such as Glue? Do you want to migrate to Unity Catalog (UC), but need help figuring out wher...
Blog Articles
Why orchestration is your key to success for modern data pipelines.
Authors: Shasidhar Eranti (@Shasidhar_ES) & Kiran Sreekumar (@Kiran_Sreekumar) In today's business landscape, data stands as the primary differentiator for companies looking to get a competitive edge ...
Databricks-to-Databricks Delta Sharing: Secure sharing across Azure regions
Delta Sharing is an open protocol developed by Databricks for secure data sharing within an organization and externally, regardless of the computing platforms used. Depending on who you are sharing th...
How to implement Slowly Changing Dimensions when you have duplicates - Part 2: Delta Live Tables
Introduction In Part 1 of this blog series, we explored the various types of duplicates, considerations for remediation, and the impacts of unchecked duplicated records on strategic decision-making. ...
Using dbt core with OAuth on Azure Databricks
By Jeroen Meulemans, Solutions Architect at Databricks Amsterdam, jeroen.meulemans@databricks.com, Thu 7 Sep 2023 Introduction This blog guides you through the process of configuring OAuth credential...
Distributed Fine Tuning of LLMs on Databricks Lakehouse with Ray AI Runtime, Part 2
Authors: Anastasia Prokaieva and Puneet Jain In our first part, we have covered the main aspects of the data loading using Hugging Face integration with the Spark dataframes and how to use RayAIR to ...
How to build operational low-latency, stateful Spark Structured Streaming pipelines
Learn to build fast, stateful pipelines for operational workloads. Discover stateless vs. stateful streams, how to setup your cluster and more. Get hands-on building a pipeline with code snippets and ...
How to Sync nested Azure AD groups to Databricks
Authors: Abhishek Pratap (@aps) & Dipankar Kushari (@dkushari)In this blog, we explore how to synchronize nested groups in Databricks from your organization’s identity provider - Azure Active Director...
Top 10 query performance tuning tips for Databricks Serverless SQL
Databricks Serverless SQL (DBSQL) is the latest offering from Databricks to build data warehouses on the Lakehouse. It incorporates all the Lakehouse features like open format, unified analytics, and ...
How SSO works with Databricks and your Identity Provider: A walkthrough
What is Single Sign-On (SSO)?The rise of Software as a Service (Saas) has proliferated the adoption of specialized software in todays modern world. This resulted in enterprises adopting many Saas serv...
-
KamLook on: Multi-region model serving on Databricks with Open...
-
-
- Khasim_1 on: Tutorial: Transform your Lakeflow Connect ad data ...
- ivanvyd on: Meet SDP Rewind: An undo button for your ETL pipel...
-
- ivanvyd on: Building Stateful Agents on Lakebase
-
- -werners- on: Identity Columns Best Practices for Databricks Lak...
- -werners- on: DABs Migration Guide: Terraform to the Direct Depl...