- 1012 Views
- 2 replies
- 7 kudos
Data Security at the level of columns or rows or Data masking
Hi everyone, I'm currently going through the Data Analyst learning path. I've just learned about Dynamic Views and I wanted to share the article on them: https://docs.databricks.com/aws/en/views/dynamic#before-you-begin There are some limitations on ...
- 1012 Views
- 2 replies
- 7 kudos
- 7 kudos
@BS_THE_ANALYST Cool stuff, right! Have read about Attribute-based Access Control (ABAC) yet? Check it out: https://docs.databricks.com/aws/en/data-governance/unity-catalog/abac/ Let me know what you think. Cheers, Louis.
- 7 kudos
- 1698 Views
- 0 replies
- 2 kudos
Databricks Asset Bundles with Python!
Databricks Asset Bundles now support Python!If you’re a Python fan, you can define jobs and pipelines in Python (or keep YAML) with DABs. You can create jobs from simple metadata, modify them during deployment with mutators, and convert existing jobs...
- 1698 Views
- 0 replies
- 2 kudos
- 4754 Views
- 0 replies
- 1 kudos
Lakehouse vs. Lakehouse Federation - Bridging the Next Evolution in Data Platforms
Over the past few years, the Lakehouse architecture has become the gold standard for managing modern data workloads. By combining the low-cost storage of data lakes with the reliability and performance of data warehouses, Lakehouses have redefined ho...
- 4754 Views
- 0 replies
- 1 kudos
- 2875 Views
- 0 replies
- 2 kudos
Understanding Modern Databricks Warehousing for the AI era: A Beginner’s Guide
IntroductionIn the current Gen AI buzz, most conversations focus on RAG for unstructured documents. But there’s another equally important challenge — making sense of structured data at scale.This is where tools like Databricks Genie step in, enablin...
- 2875 Views
- 0 replies
- 2 kudos
- 1247 Views
- 0 replies
- 0 kudos
Apache Spark 4.0 — Big Data Engineering!
The latest Spark 4.0 release delivers powerful enhancements across SQL, Python, streaming, and connectivity — all aimed at making big data workloads more efficient, reliable, and developer-friendly.With Databricks Runtime 17.0, these capabilities are...
- 1247 Views
- 0 replies
- 0 kudos
- 3392 Views
- 0 replies
- 1 kudos
Pipelines to Prompts: Getting started with Databricks and AWS
NAVIGATION:Why Data EngineeringThe Role of Data Engineering in GenAIWhat is Databricks? Unifying Data and AI on One PlatformDatabricks on AWS: A Full-Stack Platform for GenAIHands-On ExerciseFuture-Proofing: Why Data + AI Skills Matter Now More Than ...
- 3392 Views
- 0 replies
- 1 kudos
- 8704 Views
- 1 replies
- 3 kudos
Understanding Liquid Clustering in Databricks - The Next Evolution in Data Optimisation
In the world of big data, organising data smartly is just as important as collecting it. When working with large datasets in Databricks using Delta Lake, how your data is stored and ordered can greatly impact performance, especially for queries. Trad...
- 8704 Views
- 1 replies
- 3 kudos
- 3 kudos
Great post, Rahul! You’ve nailed the key trade-offs perfectly. The Appeal: LC is “set it and forget it” data management—no more manual OPTIMIZE jobs or performance firefighting. The Reality Check: Single-column clustering works great for high-cardina...
- 3 kudos
- 2359 Views
- 2 replies
- 0 kudos
Recommendations for Designing Cluster Policies Across Dev/QA/Prod Environments for DE and DA teams
Hi Community,We are working on implementing Databricks cluster policies across our organization and are seeking advice on best practices to enforce governance, security, and cost control across different environments.We have two main teams using Data...
- 2359 Views
- 2 replies
- 0 kudos
- 0 kudos
I just want to confirm one more thing here is that me as admin managing the cluster creation and no user will have access to create them me know how the cluster policies help me in this perspective.
- 0 kudos
- 5502 Views
- 4 replies
- 8 kudos
The Open Source DLT Meta Framework
DLT Meta is an open-source framework developed by Databricks Labs that enables the automation of bronze and silver data pipelines through metadata configuration rather than manual code development.At its core, the framework uses a Dataflowspec - a JS...
- 5502 Views
- 4 replies
- 8 kudos
- 7088 Views
- 0 replies
- 2 kudos
Databricks Lakeflow - Redefining Data Engineering for the Modern AI Stack
Introduction to LakeflowAt the Databricks Data + AI Summit 2025, Databricks unveiled Lakeflow, a revolutionary approach to data engineering. While many of us have used Delta Live Tables (DLT) for declarative pipeline management, Lakeflow goes beyond,...
- 7088 Views
- 0 replies
- 2 kudos
- 11014 Views
- 2 replies
- 6 kudos
Databricks LLM Evolution and Future Prospects
Databricks LLM Evolution and Future ProspectsDatabricks has progressed from a big-data compute engine to a full-stack AI powerhouse that designs, trains, and serves state‐of‐the-art large language models (LLMs). This article explores two key technica...
- 11014 Views
- 2 replies
- 6 kudos
- 6 kudos
Thanks, @RiyazAliM, for checking out the blog post!More insights on Databricks LLM and Dolly are on the way in the next one. Stay tuned and keep learning!Best,Ayush
- 6 kudos
- 9680 Views
- 3 replies
- 9 kudos
Modernizing Legacy Data Platforms to Lakehouse for AI-Readiness
As organizations increasingly migrate from legacy platforms—like on-prem SQL Server, Oracle Exadata, Teradata, Informatica, Cloudera, or Netezza—to modern cloud architectures, one critical question often arises:"Are we just lifting and shifting the s...
- 9680 Views
- 3 replies
- 9 kudos
- 9 kudos
Great article @hozefa413 , It shows all your expertise and delivery excellence
- 9 kudos
- 3166 Views
- 0 replies
- 0 kudos
Implementing data contracts on Databricks for industrial AI pipelines
Enforce schema consistency using declarative contracts on Databricks Lakehouse.Industrial AI is transforming how operations are optimized, from forecasting equipment failure to streamlining supply chains. But even the most advanced models are only as...
- 3166 Views
- 0 replies
- 0 kudos
- 4561 Views
- 2 replies
- 3 kudos
Establishing Trust relationship for Databricks on AWS
Hello.Our databricks is on Azure. We are trying to connect with AWS S3 as an external source from Unity Catalog.We have followed all steps given here, is there anything additional required?https://docs.databricks.com/aws/en/connect/unity-catalog/clou...
- 4561 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @gdschld ,what ID have you used here:"sts:ExternalId": "<STORAGE-CREDENTIAL-EXTERNAL-ID>"I haven't done this for some time and got a bit confused with this STORAGE-CREDENTIAL-EXTERNAL_ID. I used to put there Databricks Account ID.I found this, it ...
- 3 kudos
- 8693 Views
- 2 replies
- 1 kudos
How to enable unity catalog system tables?
Unity Catalog system tables provide lots of metadata & log data related to the operations of Databricks. System tables are organized into separate schemas containing one to a few tables owned and updated by Databricks. The storage and the cost of the...
- 8693 Views
- 2 replies
- 1 kudos
- 1 kudos
It's in the Databricks CLI Unity Catalog section Databricks CLI commands | Databricks DocumentationmetastoresCommands to manage metastores, which are the top-level container of objects in Unity Catalog:assign, create, current, delete, get, list, summ...
- 1 kudos
| User | Count |
|---|---|
| 85 | |
| 76 | |
| 73 | |
| 60 | |
| 43 |
