- 3612 Views
- 3 replies
- 9 kudos
Modernizing Legacy Data Platforms to Lakehouse for AI-Readiness
As organizations increasingly migrate from legacy platforms—like on-prem SQL Server, Oracle Exadata, Teradata, Informatica, Cloudera, or Netezza—to modern cloud architectures, one critical question often arises:"Are we just lifting and shifting the s...
- 3612 Views
- 3 replies
- 9 kudos
- 9 kudos
Great article @hozefa413 , It shows all your expertise and delivery excellence
- 9 kudos
- 953 Views
- 0 replies
- 0 kudos
Implementing data contracts on Databricks for industrial AI pipelines
Enforce schema consistency using declarative contracts on Databricks Lakehouse.Industrial AI is transforming how operations are optimized, from forecasting equipment failure to streamlining supply chains. But even the most advanced models are only as...
- 953 Views
- 0 replies
- 0 kudos
- 1552 Views
- 2 replies
- 3 kudos
Establishing Trust relationship for Databricks on AWS
Hello.Our databricks is on Azure. We are trying to connect with AWS S3 as an external source from Unity Catalog.We have followed all steps given here, is there anything additional required?https://docs.databricks.com/aws/en/connect/unity-catalog/clou...
- 1552 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @gdschld ,what ID have you used here:"sts:ExternalId": "<STORAGE-CREDENTIAL-EXTERNAL-ID>"I haven't done this for some time and got a bit confused with this STORAGE-CREDENTIAL-EXTERNAL_ID. I used to put there Databricks Account ID.I found this, it ...
- 3 kudos
- 4111 Views
- 0 replies
- 2 kudos
Building MultiTenant Architecture on Databricks Platform
This use case demonstrates how a SaaS product can be deployed for multiple customers or business units, ensuring data isolation at every layer through workspace separation, fine-grained access control with Unity Catalog, and secure processing using U...
- 4111 Views
- 0 replies
- 2 kudos
- 6360 Views
- 2 replies
- 1 kudos
How to enable unity catalog system tables?
Unity Catalog system tables provide lots of metadata & log data related to the operations of Databricks. System tables are organized into separate schemas containing one to a few tables owned and updated by Databricks. The storage and the cost of the...
- 6360 Views
- 2 replies
- 1 kudos
- 1 kudos
It's in the Databricks CLI Unity Catalog section Databricks CLI commands | Databricks DocumentationmetastoresCommands to manage metastores, which are the top-level container of objects in Unity Catalog:assign, create, current, delete, get, list, summ...
- 1 kudos
- 1294 Views
- 1 replies
- 3 kudos
Building DLT Pipelines with Databricks Free Edition and Amazon Q Developer
How AI-powered development accelerated my data engineering workflow Watch the Complete Development Process YouTube Video: See the entire 30-minute development sessionThis is a screen recording without voice narration showing the complete development ...
- 1294 Views
- 1 replies
- 3 kudos
- 3 kudos
This is super insightful @Pat, thanks for sharing this with the Community!
- 3 kudos
- 1027 Views
- 2 replies
- 0 kudos
Resolved! Databricks VS code extension to add cell title
I use the databricks extension in vs code for all my work. Is there any way for me to add a cell title from the extension itself?. There is no point in adding in the server version of this notebook cause when I sync the local to sever, it will overwr...
- 1027 Views
- 2 replies
- 0 kudos
- 0 kudos
One needs to use # DBTITLE 1,cell_title in a py file # COMMAND ---------- # DBTITLE 1,Title 1 from pyspark.sql import SparkSession from delta.tables import DeltaTable from pyspark.sql.functions import *
- 0 kudos
- 1011 Views
- 1 replies
- 4 kudos
The Databricks Python SDK
The Databricks SDK is a script (written in Python, in our case) which lets you control and automate actions on Databricks using the methods available in the WorkSpaceClient (more about this below).Why do we need Databricks SDK:- Automation: You can d...
- 1011 Views
- 1 replies
- 4 kudos
- 4 kudos
Good Article @RiyazAliM.
- 4 kudos
- 2300 Views
- 2 replies
- 4 kudos
Apache 4.0
Missed the Apache Spark 4.0 release? It is not just a version bump, it is a whole new level for big data processing. Some of the highlights that really stood out to me:1. SQL just got way more powerful: reusable UDFs, scripting, session variables, an...
- 2300 Views
- 2 replies
- 4 kudos
- 4 kudos
Yeah, Spark 4.0 brings powerful enhancements while staying compatible with existing workloads.Thank you for putting this together and highlighting the key updates, @ilir_nuredini.
- 4 kudos
- 8755 Views
- 3 replies
- 2 kudos
Optimizing Costs in Databricks by Dynamically Choosing Cluster Sizes
Databricks is a popular unified data analytics platform known for its powerful data processing capabilities and seamless integration with Apache Spark. However, managing and optimizing costs in Databricks can be challenging, especially when it comes ...
- 8755 Views
- 3 replies
- 2 kudos
- 2 kudos
How can this actually be used to choose a cluster pool for a Databricks workflow dynamically, that is, at run time? In other words, what can you actually do with the value of `selected_pool` other than printing it out?
- 2 kudos
- 2987 Views
- 4 replies
- 0 kudos
Data Modeling
Just got out of a session on Data Modeling using the Data Vault paradigm. Highly recommended to help think through complex data design. Look out for Data Modeling 101 for Data Lakehouse Demystified by Luan Medeiros.
- 2987 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @BS_THE_ANALYST , please use this link with code for reference :https://www.databricks.com/blog/data-vault-best-practice-implementation-lakehouse
- 0 kudos
- 1200 Views
- 0 replies
- 1 kudos
Databricks Asset Bundles
Why Should You Use Databricks Asset Bundles (DABs)?Without proper tooling, Data Engineering and Machine Learning projects can quickly become messy.That is why we recommend leveraging DABs to solve these common challenges:1. Collaboration:Without stru...
- 1200 Views
- 0 replies
- 1 kudos
- 9049 Views
- 11 replies
- 41 kudos
From Associate to Professional: My Learning Plan to ace all Databricks Data Engineer Certifications
In today’s data-driven world, the role of a data engineer is critical in designing and maintaining the infrastructure that allows for the efficient collection, storage, and analysis of large volumes of data. Databricks certifications holds significan...
- 9049 Views
- 11 replies
- 41 kudos
- 41 kudos
@SumitSingh this is getting put in the favourites. Thanks a bunch for this All the best,BS
- 41 kudos
- 8562 Views
- 8 replies
- 8 kudos
My Journey with Schema Management in Databricks
When I first started handling schema management in Databricks, I realized that a little bit of planning could save me a lot of headaches down the road. Here’s what I’ve learned and some simple tips that helped me manage schema changes effectively. On...
- 8562 Views
- 8 replies
- 8 kudos
- 8 kudos
Haha, glad it made sense! Joao.Try it out, and if you run into any issues, just let me know. Always happy to help! And best friends? You got it!
- 8 kudos
- 1274 Views
- 2 replies
- 6 kudos
🔐 How Do I Prevent Users from Accidentally Deleting Tables in Unity Catalog? 🔐
Question:I have a role called dev-dataengineer with the following privileges on the catalog dap_catalog_dev:APPLY TAGCREATE FUNCTIONCREATE MATERIALIZED VIEWCREATE TABLECREATE VOLUMEEXECUTEREAD VOLUMEREFRESHSELECTUSE SCHEMAWRITE VOLUMEDespite this, u...
- 1274 Views
- 2 replies
- 6 kudos
- 6 kudos
Managing assets in UC is always a overhead maintenance. We have this access controls in terraform codes and it is always hard to see what level of access is given to different personas in the org. We are building an audit dashboard for it.
- 6 kudos
