A Key challenge for Organizations is to ensure that data metrics refer to the same for all teams. If BI logic is scattered across various tools, SQL and notebooks, a metrics tax is levied (multiple dashboards showing different revenues). Databricks M...
In financial services, data never sleeps. Trades flow in every second. Risk calculations refresh continuously. Regulatory reports demand precision. BI dashboards serve business users who expect sub-second responses. And behind all of that? A massive ...
@Nidhi_Patni Thanks for production level information!@wesleyfelipe > I have one question: what was the trade-off between using traditional partitioning with Z-ordering versus liquid clustering?Traditional partitioning with Z-ordering is kind of the o...
Hi,Is there a way to custom name an assetbundle file name and pass that to databricks bundle deploy?I mean right now I must use databricks.yml, so my question is whether there is a way to pass a custom file name.note that I don't want to embed a file...
Hi @murtadha_s The simple answer is no, but I’d like to understand the issue you’re facing so I can see if there’s anything I can help with.For example, this is how I’m using it in my production application, and it’s working quite well. We’re handlin...
When working with externally managed Delta tables and traditional partitioning strategies (for example by day, week, or month), one common challenge is:How large are my partitions actually?Before deciding whether to partition by day vs. week vs. mont...
AI didn’t fail to replace developers, It exposed something deeper.In the rush to adopt AI, many organizations assumed that coding assistants and automation would reduce engineering effort and accelerate delivery. But the reality has been different. M...
AI/BI dashboards can now be managed through Terraform.Dashboard using serialized_dashboard attribute: data "databricks_sql_warehouse" "starter" { name = "Starter Warehouse" } resource "databricks_dashboard" "dashboard" { display_name = "...
if the content of `file_path` has changed, terraform detects no changes. It is better if you make it checking md5 of the file to allow resource updates.
In a Databricks project integrating multiple legacy systems, one recurring challenge was maintaining development consistency as pipelines and team size grew.Pipeline divergence tends to emerge quickly:• Different ingestion approaches• Inconsistent tr...
Organizations created report factories in the last few decades. We have spent decades in creating the paginated reports and interactive dashboards in various BI tools yet face the same bottleneck when the business asks a question, and data team takes...
What an excellent evening at the inaugural Databricks Bristol Meetup! It was great to finally have an Databricks community in Bristol. A massive thanks to iO Associates for pulling this together and also to the awesome speakers. The evening had two g...
Please check https://github.com/rsleedbx/crdb_to_dbx which has the steps and a working notebook. This guide shows how to stream CockroachDB data to Databricks using changefeeds, Azure Blob Storage, Unity Catalog, and Delta Lake. You get one platform...
Introduction“AI First” - But Data Always Comes FirstI have been working in the data space for close to two decades. My journey started as an ETL developer and gradually evolved into roles spanning data engineering, platform design, and solution archi...
@Saurabh2406 , I really appreciate how you grounded the “AI-first” conversation in the reality that data governance, security, and quality are what actually determine whether AI can scale beyond pilots. The tie-in to Gartner’s AI maturity model, and ...
Wait, Did Databricks Just Put Git Inside My Database? If you've been scratching your head at Lakebase's "branching" feature wondering "am I working with a database or GitHub?"—you're not alone. Let me break down what's actually happening here, becaus...
@AbhaySingh , This was a fun read — and a great way to spark discussion about what “Git inside my database” really means in practice. From what I’m seeing in the product world, Databricks isn’t literally putting Git inside the storage engine of your...
IntroductionCloud-native data platforms like Azure Databricks are powerful because they abstract away infrastructure so you can focus on data engineering, analytics, and ML workloads. However, there are situations where you may run into issues that r...
Databricks has added 2 new feature on its UI. These are small but quite effective for the developer productivity. 1. Paste images into notebooksCopy images from your local file system and paste them into markdown cells in Databricks notebookshttps:/...
IntroductionScaling data pipelines across an organization can be challenging, particularly when data sources, requirements, and transformation rules are always changing. A metadata table-driven framework using LakeFlow Declarative (Formerly DLT) enab...
can you please share the details how this can be implemented using a sample use case in step by step process. Also python code that needs to written in each layer (bronze/silver/gold)
| User | Count |
|---|---|
| 85 | |
| 74 | |
| 59 | |
| 44 | |
| 44 |
