cancel
Showing results for 
Search instead for 
Did you mean: 
Community Articles
Dive into a collaborative space where members like YOU can exchange knowledge, tips, and best practices. Join the conversation today and unlock a wealth of collective wisdom to enhance your experience and drive success.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

devipriya
by New Contributor III
  • 2574 Views
  • 0 replies
  • 1 kudos

Pipelines to Prompts: Getting started with Databricks and AWS

NAVIGATION:Why Data EngineeringThe Role of Data Engineering in GenAIWhat is Databricks? Unifying Data and AI on One PlatformDatabricks on AWS: A Full-Stack Platform for GenAIHands-On ExerciseFuture-Proofing: Why Data + AI Skills Matter Now More Than ...

Luluspace34_0-1754719688416.png Luluspace34_1-1754719687902.png Luluspace34_2-1754719687532.png Luluspace34_3-1754719688664.png
  • 2574 Views
  • 0 replies
  • 1 kudos
RahulGupta
by New Contributor III
  • 1381 Views
  • 1 replies
  • 3 kudos

Understanding Liquid Clustering in Databricks - The Next Evolution in Data Optimisation

In the world of big data, organising data smartly is just as important as collecting it. When working with large datasets in Databricks using Delta Lake, how your data is stored and ordered can greatly impact performance, especially for queries. Trad...

  • 1381 Views
  • 1 replies
  • 3 kudos
Latest Reply
BigRoux
Databricks Employee
  • 3 kudos

Great post, Rahul! You’ve nailed the key trade-offs perfectly. The Appeal: LC is “set it and forget it” data management—no more manual OPTIMIZE jobs or performance firefighting. The Reality Check: Single-column clustering works great for high-cardina...

  • 3 kudos
Charansai
by New Contributor III
  • 768 Views
  • 2 replies
  • 0 kudos

Recommendations for Designing Cluster Policies Across Dev/QA/Prod Environments for DE and DA teams

Hi Community,We are working on implementing Databricks cluster policies across our organization and are seeking advice on best practices to enforce governance, security, and cost control across different environments.We have two main teams using Data...

  • 768 Views
  • 2 replies
  • 0 kudos
Latest Reply
Charansai
New Contributor III
  • 0 kudos

I just want to confirm one more thing here is that me as admin managing the cluster creation and no user will have access to create them me know how the cluster policies help me in this perspective.

  • 0 kudos
1 More Replies
RiyazAliM
by Honored Contributor
  • 1777 Views
  • 4 replies
  • 7 kudos

The Open Source DLT Meta Framework

DLT Meta is an open-source framework developed by Databricks Labs that enables the automation of bronze and silver data pipelines through metadata configuration rather than manual code development.At its core, the framework uses a Dataflowspec - a JS...

aayrm5_1-1753154791734.png aayrm5_3-1753155396659.png aayrm5_4-1753155477157.png aayrm5_2-1753155019763.png
  • 1777 Views
  • 4 replies
  • 7 kudos
Latest Reply
sridharplv
Valued Contributor II
  • 7 kudos

Great Article Riyaz. keep Sharing more knowledge

  • 7 kudos
3 More Replies
Ajay-Pandey
by Esteemed Contributor III
  • 4158 Views
  • 3 replies
  • 5 kudos

Cross-filtering for AI/BI dashboards

AI/BI dashboards now support cross-filtering, which allows you to click on an element in one chart to filter and update related data in other charts.Cross-filtering allows users to interactively explore relationships and patterns across multiple visu...

cross-filter.gif
Community Articles
AI BI
Cross-filtering
Dashboards
reports
  • 4158 Views
  • 3 replies
  • 5 kudos
Latest Reply
SFDataEng
New Contributor III
  • 5 kudos

Yes, something like that could do. Reflections on the video/concept/comment are below.Summary:Option 1: Upon adding a cross-filter, it could work to add a short, top horizontal bar of cross-filters (which expands vertically if there are more than fit...

  • 5 kudos
2 More Replies
Nidhi_Patni
by New Contributor III
  • 1219 Views
  • 2 replies
  • 5 kudos

How We Built Robust Data Governance at Scale

In today's data-driven world, trust is currency—and that trust starts with quality data governed by strong principles. For one of our client, where we're on a mission to build intelligent enterprises with AI, data isn't just an asset—it's a responsib...

Nidhi_Patni_0-1753460966300.png Nidhi_Patni_5-1753461715636.png Nidhi_Patni_2-1753460966324.png Nidhi_Patni_3-1753460966330.png
  • 1219 Views
  • 2 replies
  • 5 kudos
Latest Reply
Dr-Sylvester
New Contributor III
  • 5 kudos

Great Article  @Nidhi_Patni 

  • 5 kudos
1 More Replies
ayushbadhera1
by New Contributor III
  • 890 Views
  • 2 replies
  • 6 kudos

Databricks LLM Evolution and Future Prospects

Databricks LLM Evolution and Future ProspectsDatabricks has progressed from a big-data compute engine to a full-stack AI powerhouse that designs, trains, and serves state‐of‐the-art large language models (LLMs). This article explores two key technica...

  • 890 Views
  • 2 replies
  • 6 kudos
Latest Reply
ayushbadhera1
New Contributor III
  • 6 kudos

Thanks, @RiyazAliM, for checking out the blog post!More insights on Databricks LLM and Dolly are on the way in the next one. Stay tuned and keep learning!Best,Ayush

  • 6 kudos
1 More Replies
TejeshS
by Contributor
  • 1326 Views
  • 2 replies
  • 7 kudos

Building a Metadata Table-Driven Framework Using LakeFlow Declarative (Formerly DLT) Pipelines

IntroductionScaling data pipelines across an organization can be challenging, particularly when data sources, requirements, and transformation rules are always changing. A metadata table-driven framework using LakeFlow Declarative (Formerly DLT) enab...

TejeshS_0-1753177497326.png
  • 1326 Views
  • 2 replies
  • 7 kudos
Latest Reply
sridharplv
Valued Contributor II
  • 7 kudos

Good one Tejesh. Quick intro on DLT meta.

  • 7 kudos
1 More Replies
hozefa413
by New Contributor III
  • 2062 Views
  • 3 replies
  • 9 kudos

Modernizing Legacy Data Platforms to Lakehouse for AI-Readiness

As organizations increasingly migrate from legacy platforms—like on-prem SQL Server, Oracle Exadata, Teradata, Informatica, Cloudera, or Netezza—to modern cloud architectures, one critical question often arises:"Are we just lifting and shifting the s...

  • 2062 Views
  • 3 replies
  • 9 kudos
Latest Reply
sridharplv
Valued Contributor II
  • 9 kudos

Great article @hozefa413 , It shows all your expertise and delivery excellence

  • 9 kudos
2 More Replies
gdschld
by New Contributor
  • 1001 Views
  • 2 replies
  • 3 kudos

Establishing Trust relationship for Databricks on AWS

Hello.Our databricks is on Azure. We are trying to connect with AWS S3 as an external source from Unity Catalog.We have followed all steps given here, is there anything additional required?https://docs.databricks.com/aws/en/connect/unity-catalog/clou...

  • 1001 Views
  • 2 replies
  • 3 kudos
Latest Reply
Pat
Esteemed Contributor
  • 3 kudos

Hi @gdschld ,what ID have you used here:"sts:ExternalId": "<STORAGE-CREDENTIAL-EXTERNAL-ID>"I haven't done this for some time and got a bit confused with this STORAGE-CREDENTIAL-EXTERNAL_ID. I used to put there Databricks Account ID.I found this, it ...

  • 3 kudos
1 More Replies
DouglasMoore
by Databricks Employee
  • 5536 Views
  • 2 replies
  • 1 kudos

How to enable unity catalog system tables?

Unity Catalog system tables provide lots of metadata & log data related to the operations of Databricks. System tables are organized into separate schemas containing one to a few tables owned and updated by Databricks. The storage and the cost of the...

  • 5536 Views
  • 2 replies
  • 1 kudos
Latest Reply
AlanD
New Contributor III
  • 1 kudos

It's in the Databricks CLI Unity Catalog section Databricks CLI commands | Databricks DocumentationmetastoresCommands to manage metastores, which are the top-level container of objects in Unity Catalog:assign, create, current, delete, get, list, summ...

  • 1 kudos
1 More Replies
Pat
by Esteemed Contributor
  • 838 Views
  • 1 replies
  • 3 kudos

Building DLT Pipelines with Databricks Free Edition and Amazon Q Developer

How AI-powered development accelerated my data engineering workflow Watch the Complete Development Process YouTube Video: See the entire 30-minute development sessionThis is a screen recording without voice narration showing the complete development ...

  • 838 Views
  • 1 replies
  • 3 kudos
Latest Reply
Advika
Databricks Employee
  • 3 kudos

This is super insightful @Pat, thanks for sharing this with the Community!

  • 3 kudos

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels