- 434 Views
- 1 replies
- 0 kudos
Resolved! Automating Notebook Documentation in Databricks with LLMs
In one of my projects, I needed to generate structured documentation for an entire directory of Databricks notebooks.This solution uses the Databricks Workspace API together with a Serving Endpoint (LLM) to automatically create HTML documentation for...
- 434 Views
- 1 replies
- 0 kudos
- 0 kudos
Suggestions are always welcome — I hope this helps anyone looking to automate notebook documentation in Databricks.
- 0 kudos
- 283 Views
- 2 replies
- 7 kudos
Data Security at the level of columns or rows or Data masking
Hi everyone, I'm currently going through the Data Analyst learning path. I've just learned about Dynamic Views and I wanted to share the article on them: https://docs.databricks.com/aws/en/views/dynamic#before-you-begin There are some limitations on ...
- 283 Views
- 2 replies
- 7 kudos
- 7 kudos
@BS_THE_ANALYST Cool stuff, right! Have read about Attribute-based Access Control (ABAC) yet? Check it out: https://docs.databricks.com/aws/en/data-governance/unity-catalog/abac/ Let me know what you think. Cheers, Louis.
- 7 kudos
- 174 Views
- 0 replies
- 2 kudos
Databricks Asset Bundles with Python!
Databricks Asset Bundles now support Python!If you’re a Python fan, you can define jobs and pipelines in Python (or keep YAML) with DABs. You can create jobs from simple metadata, modify them during deployment with mutators, and convert existing jobs...
- 174 Views
- 0 replies
- 2 kudos
- 607 Views
- 2 replies
- 3 kudos
Getting Databricks Generative AI Engineer Associate (and What I Learned)
I just earned my Databricks Certified Generative AI Engineer Associate Certification, and in this post, I’m sharing the key tips, resources, and personal insights that helped me succeed. My certiticate from DatabricksNavigation:PreludeAbout the certR...
- 607 Views
- 2 replies
- 3 kudos
- 3 kudos
Thank you @devipriya for providing the detailed information on certifications.
- 3 kudos
- 329 Views
- 0 replies
- 1 kudos
Lakehouse vs. Lakehouse Federation - Bridging the Next Evolution in Data Platforms
Over the past few years, the Lakehouse architecture has become the gold standard for managing modern data workloads. By combining the low-cost storage of data lakes with the reliability and performance of data warehouses, Lakehouses have redefined ho...
- 329 Views
- 0 replies
- 1 kudos
- 385 Views
- 0 replies
- 2 kudos
Understanding Modern Databricks Warehousing for the AI era: A Beginner’s Guide
IntroductionIn the current Gen AI buzz, most conversations focus on RAG for unstructured documents. But there’s another equally important challenge — making sense of structured data at scale.This is where tools like Databricks Genie step in, enablin...
- 385 Views
- 0 replies
- 2 kudos
- 285 Views
- 0 replies
- 0 kudos
Apache Spark 4.0 — Big Data Engineering!
The latest Spark 4.0 release delivers powerful enhancements across SQL, Python, streaming, and connectivity — all aimed at making big data workloads more efficient, reliable, and developer-friendly.With Databricks Runtime 17.0, these capabilities are...
- 285 Views
- 0 replies
- 0 kudos
- 403 Views
- 0 replies
- 2 kudos
Databricks AI/BI Genie: The Future of Conversational Analytics
The Rise of AI in Data AnalyticsOver the last decade, organizations have collected massive amounts of data from customer transactions to IoT sensors, web logs, and financial records. But collecting data is just the first step. The real challenge lies...
- 403 Views
- 0 replies
- 2 kudos
- 2529 Views
- 0 replies
- 1 kudos
Pipelines to Prompts: Getting started with Databricks and AWS
NAVIGATION:Why Data EngineeringThe Role of Data Engineering in GenAIWhat is Databricks? Unifying Data and AI on One PlatformDatabricks on AWS: A Full-Stack Platform for GenAIHands-On ExerciseFuture-Proofing: Why Data + AI Skills Matter Now More Than ...
- 2529 Views
- 0 replies
- 1 kudos
- 1284 Views
- 1 replies
- 3 kudos
Understanding Liquid Clustering in Databricks - The Next Evolution in Data Optimisation
In the world of big data, organising data smartly is just as important as collecting it. When working with large datasets in Databricks using Delta Lake, how your data is stored and ordered can greatly impact performance, especially for queries. Trad...
- 1284 Views
- 1 replies
- 3 kudos
- 3 kudos
Great post, Rahul! You’ve nailed the key trade-offs perfectly. The Appeal: LC is “set it and forget it” data management—no more manual OPTIMIZE jobs or performance firefighting. The Reality Check: Single-column clustering works great for high-cardina...
- 3 kudos
- 745 Views
- 2 replies
- 0 kudos
Recommendations for Designing Cluster Policies Across Dev/QA/Prod Environments for DE and DA teams
Hi Community,We are working on implementing Databricks cluster policies across our organization and are seeking advice on best practices to enforce governance, security, and cost control across different environments.We have two main teams using Data...
- 745 Views
- 2 replies
- 0 kudos
- 0 kudos
I just want to confirm one more thing here is that me as admin managing the cluster creation and no user will have access to create them me know how the cluster policies help me in this perspective.
- 0 kudos
- 1684 Views
- 4 replies
- 7 kudos
The Open Source DLT Meta Framework
DLT Meta is an open-source framework developed by Databricks Labs that enables the automation of bronze and silver data pipelines through metadata configuration rather than manual code development.At its core, the framework uses a Dataflowspec - a JS...
- 1684 Views
- 4 replies
- 7 kudos
- 4035 Views
- 3 replies
- 5 kudos
Cross-filtering for AI/BI dashboards
AI/BI dashboards now support cross-filtering, which allows you to click on an element in one chart to filter and update related data in other charts.Cross-filtering allows users to interactively explore relationships and patterns across multiple visu...
- 4035 Views
- 3 replies
- 5 kudos
- 5 kudos
Yes, something like that could do. Reflections on the video/concept/comment are below.Summary:Option 1: Upon adding a cross-filter, it could work to add a short, top horizontal bar of cross-filters (which expands vertically if there are more than fit...
- 5 kudos
- 1151 Views
- 2 replies
- 5 kudos
How We Built Robust Data Governance at Scale
In today's data-driven world, trust is currency—and that trust starts with quality data governed by strong principles. For one of our client, where we're on a mission to build intelligent enterprises with AI, data isn't just an asset—it's a responsib...
- 1151 Views
- 2 replies
- 5 kudos
- 3429 Views
- 0 replies
- 2 kudos
Databricks Lakeflow - Redefining Data Engineering for the Modern AI Stack
Introduction to LakeflowAt the Databricks Data + AI Summit 2025, Databricks unveiled Lakeflow, a revolutionary approach to data engineering. While many of us have used Delta Live Tables (DLT) for declarative pipeline management, Lakeflow goes beyond,...
- 3429 Views
- 0 replies
- 2 kudos
