cancel
Showing results for 
Search instead for 
Did you mean: 
Community Articles
Dive into a collaborative space where members like YOU can exchange knowledge, tips, and best practices. Join the conversation today and unlock a wealth of collective wisdom to enhance your experience and drive success.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

devipriya
by New Contributor III
  • 635 Views
  • 0 replies
  • 2 kudos

Understanding Modern Databricks Warehousing for the AI era: A Beginner’s Guide

 IntroductionIn the current Gen AI buzz, most conversations focus on RAG for unstructured documents. But there’s another equally important challenge — making sense of structured data at scale.This is where tools like Databricks Genie step in, enablin...

devipriya_0-1755110417311.png devipriya_1-1755110416034.png devipriya_2-1755110416212.png devipriya_3-1755110416450.png
  • 635 Views
  • 0 replies
  • 2 kudos
DILIPKHANDELWAL
by New Contributor
  • 377 Views
  • 0 replies
  • 0 kudos

Apache Spark 4.0 — Big Data Engineering!

The latest Spark 4.0 release delivers powerful enhancements across SQL, Python, streaming, and connectivity — all aimed at making big data workloads more efficient, reliable, and developer-friendly.With Databricks Runtime 17.0, these capabilities are...

  • 377 Views
  • 0 replies
  • 0 kudos
devipriya
by New Contributor III
  • 2669 Views
  • 0 replies
  • 1 kudos

Pipelines to Prompts: Getting started with Databricks and AWS

NAVIGATION:Why Data EngineeringThe Role of Data Engineering in GenAIWhat is Databricks? Unifying Data and AI on One PlatformDatabricks on AWS: A Full-Stack Platform for GenAIHands-On ExerciseFuture-Proofing: Why Data + AI Skills Matter Now More Than ...

Luluspace34_0-1754719688416.png Luluspace34_1-1754719687902.png Luluspace34_2-1754719687532.png Luluspace34_3-1754719688664.png
  • 2669 Views
  • 0 replies
  • 1 kudos
RahulGupta
by New Contributor III
  • 1703 Views
  • 1 replies
  • 3 kudos

Understanding Liquid Clustering in Databricks - The Next Evolution in Data Optimisation

In the world of big data, organising data smartly is just as important as collecting it. When working with large datasets in Databricks using Delta Lake, how your data is stored and ordered can greatly impact performance, especially for queries. Trad...

  • 1703 Views
  • 1 replies
  • 3 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 3 kudos

Great post, Rahul! You’ve nailed the key trade-offs perfectly. The Appeal: LC is “set it and forget it” data management—no more manual OPTIMIZE jobs or performance firefighting. The Reality Check: Single-column clustering works great for high-cardina...

  • 3 kudos
Charansai
by New Contributor III
  • 898 Views
  • 2 replies
  • 0 kudos

Recommendations for Designing Cluster Policies Across Dev/QA/Prod Environments for DE and DA teams

Hi Community,We are working on implementing Databricks cluster policies across our organization and are seeking advice on best practices to enforce governance, security, and cost control across different environments.We have two main teams using Data...

  • 898 Views
  • 2 replies
  • 0 kudos
Latest Reply
Charansai
New Contributor III
  • 0 kudos

I just want to confirm one more thing here is that me as admin managing the cluster creation and no user will have access to create them me know how the cluster policies help me in this perspective.

  • 0 kudos
1 More Replies
RiyazAliM
by Honored Contributor
  • 2126 Views
  • 4 replies
  • 7 kudos

The Open Source DLT Meta Framework

DLT Meta is an open-source framework developed by Databricks Labs that enables the automation of bronze and silver data pipelines through metadata configuration rather than manual code development.At its core, the framework uses a Dataflowspec - a JS...

aayrm5_1-1753154791734.png aayrm5_3-1753155396659.png aayrm5_4-1753155477157.png aayrm5_2-1753155019763.png
  • 2126 Views
  • 4 replies
  • 7 kudos
Latest Reply
sridharplv
Valued Contributor II
  • 7 kudos

Great Article Riyaz. keep Sharing more knowledge

  • 7 kudos
3 More Replies
Nidhi_Patni
by New Contributor III
  • 1497 Views
  • 2 replies
  • 5 kudos

How We Built Robust Data Governance at Scale

In today's data-driven world, trust is currency—and that trust starts with quality data governed by strong principles. For one of our client, where we're on a mission to build intelligent enterprises with AI, data isn't just an asset—it's a responsib...

Nidhi_Patni_0-1753460966300.png Nidhi_Patni_5-1753461715636.png Nidhi_Patni_2-1753460966324.png Nidhi_Patni_3-1753460966330.png
  • 1497 Views
  • 2 replies
  • 5 kudos
Latest Reply
Dr-Sylvester
New Contributor III
  • 5 kudos

Great Article  @Nidhi_Patni 

  • 5 kudos
1 More Replies
ayushbadhera1
by New Contributor III
  • 1076 Views
  • 2 replies
  • 6 kudos

Databricks LLM Evolution and Future Prospects

Databricks LLM Evolution and Future ProspectsDatabricks has progressed from a big-data compute engine to a full-stack AI powerhouse that designs, trains, and serves state‐of‐the-art large language models (LLMs). This article explores two key technica...

  • 1076 Views
  • 2 replies
  • 6 kudos
Latest Reply
ayushbadhera1
New Contributor III
  • 6 kudos

Thanks, @RiyazAliM, for checking out the blog post!More insights on Databricks LLM and Dolly are on the way in the next one. Stay tuned and keep learning!Best,Ayush

  • 6 kudos
1 More Replies
hozefa413
by New Contributor III
  • 2537 Views
  • 3 replies
  • 9 kudos

Modernizing Legacy Data Platforms to Lakehouse for AI-Readiness

As organizations increasingly migrate from legacy platforms—like on-prem SQL Server, Oracle Exadata, Teradata, Informatica, Cloudera, or Netezza—to modern cloud architectures, one critical question often arises:"Are we just lifting and shifting the s...

  • 2537 Views
  • 3 replies
  • 9 kudos
Latest Reply
sridharplv
Valued Contributor II
  • 9 kudos

Great article @hozefa413 , It shows all your expertise and delivery excellence

  • 9 kudos
2 More Replies
gdschld
by New Contributor
  • 1242 Views
  • 2 replies
  • 3 kudos

Establishing Trust relationship for Databricks on AWS

Hello.Our databricks is on Azure. We are trying to connect with AWS S3 as an external source from Unity Catalog.We have followed all steps given here, is there anything additional required?https://docs.databricks.com/aws/en/connect/unity-catalog/clou...

  • 1242 Views
  • 2 replies
  • 3 kudos
Latest Reply
Pat
Esteemed Contributor
  • 3 kudos

Hi @gdschld ,what ID have you used here:"sts:ExternalId": "<STORAGE-CREDENTIAL-EXTERNAL-ID>"I haven't done this for some time and got a bit confused with this STORAGE-CREDENTIAL-EXTERNAL_ID. I used to put there Databricks Account ID.I found this, it ...

  • 3 kudos
1 More Replies
DouglasMoore
by Databricks Employee
  • 5908 Views
  • 2 replies
  • 1 kudos

How to enable unity catalog system tables?

Unity Catalog system tables provide lots of metadata & log data related to the operations of Databricks. System tables are organized into separate schemas containing one to a few tables owned and updated by Databricks. The storage and the cost of the...

  • 5908 Views
  • 2 replies
  • 1 kudos
Latest Reply
AlanD
New Contributor III
  • 1 kudos

It's in the Databricks CLI Unity Catalog section Databricks CLI commands | Databricks DocumentationmetastoresCommands to manage metastores, which are the top-level container of objects in Unity Catalog:assign, create, current, delete, get, list, summ...

  • 1 kudos
1 More Replies
Pat
by Esteemed Contributor
  • 1074 Views
  • 1 replies
  • 3 kudos

Building DLT Pipelines with Databricks Free Edition and Amazon Q Developer

How AI-powered development accelerated my data engineering workflow Watch the Complete Development Process YouTube Video: See the entire 30-minute development sessionThis is a screen recording without voice narration showing the complete development ...

  • 1074 Views
  • 1 replies
  • 3 kudos
Latest Reply
Advika
Databricks Employee
  • 3 kudos

This is super insightful @Pat, thanks for sharing this with the Community!

  • 3 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels