- 6318 Views
- 11 replies
- 39 kudos
From Associate to Professional: My Learning Plan to ace all Databricks Data Engineer Certifications
In today’s data-driven world, the role of a data engineer is critical in designing and maintaining the infrastructure that allows for the efficient collection, storage, and analysis of large volumes of data. Databricks certifications holds significan...
- 6318 Views
- 11 replies
- 39 kudos
- 39 kudos
@SumitSingh this is getting put in the favourites. Thanks a bunch for this All the best,BS
- 39 kudos
- 61 Views
- 2 replies
- 4 kudos
Resolved! Databricks for RAG: Build, Run, Evaluate
What is RAG?RAG (Retrieval-Augmented Generation) on Databricks refers to building and running AI applications that combine:Retrieval systems (like vector databases or search over documents)Generative AI models (such as LLMs like GPT)within the Databr...
- 61 Views
- 2 replies
- 4 kudos
- 662 Views
- 1 replies
- 3 kudos
Understanding Liquid Clustering in Databricks - The Next Evolution in Data Optimisation
In the world of big data, organising data smartly is just as important as collecting it. When working with large datasets in Databricks using Delta Lake, how your data is stored and ordered can greatly impact performance, especially for queries. Trad...
- 662 Views
- 1 replies
- 3 kudos
- 3 kudos
Great post, Rahul! You’ve nailed the key trade-offs perfectly. The Appeal: LC is “set it and forget it” data management—no more manual OPTIMIZE jobs or performance firefighting. The Reality Check: Single-column clustering works great for high-cardina...
- 3 kudos
- 488 Views
- 2 replies
- 0 kudos
Recommendations for Designing Cluster Policies Across Dev/QA/Prod Environments for DE and DA teams
Hi Community,We are working on implementing Databricks cluster policies across our organization and are seeking advice on best practices to enforce governance, security, and cost control across different environments.We have two main teams using Data...
- 488 Views
- 2 replies
- 0 kudos
- 0 kudos
I just want to confirm one more thing here is that me as admin managing the cluster creation and no user will have access to create them me know how the cluster policies help me in this perspective.
- 0 kudos
- 1020 Views
- 4 replies
- 7 kudos
The Open Source DLT Meta Framework
DLT Meta is an open-source framework developed by Databricks Labs that enables the automation of bronze and silver data pipelines through metadata configuration rather than manual code development.At its core, the framework uses a Dataflowspec - a JS...
- 1020 Views
- 4 replies
- 7 kudos
- 589 Views
- 2 replies
- 7 kudos
Attribute-Based Access Control (ABAC) in Databricks Unity Catalog
What Is ABAC and Why Does It Matter?Attribute-Based Access Control (ABAC) is a data governance model now available in Databricks, designed to offer fine-grained, dynamic, and scalable access control for data, AI assets, and files managed through Data...
- 589 Views
- 2 replies
- 7 kudos
- 3415 Views
- 3 replies
- 5 kudos
Cross-filtering for AI/BI dashboards
AI/BI dashboards now support cross-filtering, which allows you to click on an element in one chart to filter and update related data in other charts.Cross-filtering allows users to interactively explore relationships and patterns across multiple visu...
- 3415 Views
- 3 replies
- 5 kudos
- 5 kudos
Yes, something like that could do. Reflections on the video/concept/comment are below.Summary:Option 1: Upon adding a cross-filter, it could work to add a short, top horizontal bar of cross-filters (which expands vertically if there are more than fit...
- 5 kudos
- 841 Views
- 2 replies
- 2 kudos
Demo Deploy a Databricks Asset Bundle with Azure DevOps Pipelines
I recently wrote a blog post showing how to deploy a minimal Databricks job with notebook task using Databricks Asset Bundles and Azure DevOps pipelines. The code is available in my public GitHub repo. Thought I'd share it here!ObjectivesIn this post...
- 841 Views
- 2 replies
- 2 kudos
- 567 Views
- 1 replies
- 6 kudos
From Pilots to Production: Unlocking Enterprise AI Agents with Agent Bricks
Many enterprises today launch AI agents with high hopes but more often than not, those pilots never reach production. The culprit? Complexity, poor evaluations, ballooning costs, and governance gaps.Why do so many AI agent pilots never make it to pro...
- 567 Views
- 1 replies
- 6 kudos
- 6 kudos
Summed up so clearly, @Purvansh!
- 6 kudos
- 760 Views
- 2 replies
- 5 kudos
How We Built Robust Data Governance at Scale
In today's data-driven world, trust is currency—and that trust starts with quality data governed by strong principles. For one of our client, where we're on a mission to build intelligent enterprises with AI, data isn't just an asset—it's a responsib...
- 760 Views
- 2 replies
- 5 kudos
- 915 Views
- 0 replies
- 1 kudos
Databricks Lakeflow - Redefining Data Engineering for the Modern AI Stack
Introduction to LakeflowAt the Databricks Data + AI Summit 2025, Databricks unveiled Lakeflow, a revolutionary approach to data engineering. While many of us have used Delta Live Tables (DLT) for declarative pipeline management, Lakeflow goes beyond,...
- 915 Views
- 0 replies
- 1 kudos
- 536 Views
- 2 replies
- 5 kudos
Databricks LLM Evolution and Future Prospects
Databricks LLM Evolution and Future ProspectsDatabricks has progressed from a big-data compute engine to a full-stack AI powerhouse that designs, trains, and serves state‐of‐the-art large language models (LLMs). This article explores two key technica...
- 536 Views
- 2 replies
- 5 kudos
- 5 kudos
Thanks, @RiyazAliM, for checking out the blog post!More insights on Databricks LLM and Dolly are on the way in the next one. Stay tuned and keep learning!Best,Ayush
- 5 kudos
- 565 Views
- 2 replies
- 7 kudos
Building a Metadata Table-Driven Framework Using LakeFlow Declarative (Formerly DLT) Pipelines
IntroductionScaling data pipelines across an organization can be challenging, particularly when data sources, requirements, and transformation rules are always changing. A metadata table-driven framework using LakeFlow Declarative (Formerly DLT) enab...
- 565 Views
- 2 replies
- 7 kudos
- 851 Views
- 3 replies
- 9 kudos
Modernizing Legacy Data Platforms to Lakehouse for AI-Readiness
As organizations increasingly migrate from legacy platforms—like on-prem SQL Server, Oracle Exadata, Teradata, Informatica, Cloudera, or Netezza—to modern cloud architectures, one critical question often arises:"Are we just lifting and shifting the s...
- 851 Views
- 3 replies
- 9 kudos
- 9 kudos
Great article @hozefa413 , It shows all your expertise and delivery excellence
- 9 kudos
- 280 Views
- 0 replies
- 0 kudos
Implementing data contracts on Databricks for industrial AI pipelines
Enforce schema consistency using declarative contracts on Databricks Lakehouse.Industrial AI is transforming how operations are optimized, from forecasting equipment failure to streamlining supply chains. But even the most advanced models are only as...
- 280 Views
- 0 replies
- 0 kudos
- 624 Views
- 2 replies
- 3 kudos
Establishing Trust relationship for Databricks on AWS
Hello.Our databricks is on Azure. We are trying to connect with AWS S3 as an external source from Unity Catalog.We have followed all steps given here, is there anything additional required?https://docs.databricks.com/aws/en/connect/unity-catalog/clou...
- 624 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @gdschld ,what ID have you used here:"sts:ExternalId": "<STORAGE-CREDENTIAL-EXTERNAL-ID>"I haven't done this for some time and got a bit confused with this STORAGE-CREDENTIAL-EXTERNAL_ID. I used to put there Databricks Account ID.I found this, it ...
- 3 kudos
