The Real Problem: Kafka Source Parallelism in SparkBefore discussing foreachBatch, multi-table writes, or any specific use case, it helps to understand the underlying issue. This is a problem with how Spark Structured Streaming consumes from Kafka, a...
How to handle MERGE with Schema Evolution in Delta LakeHi everyone,Schema evolution during MERGE is one of the trickiest parts of building robust Delta Lake pipelines. Databricks actually has a native SQL syntax for this — plus Python API options for...
Great post. Would also like to consider the following points:Guardrails: schema evolution is powerful — it can also accidentally add garbage columns if upstream sends unexpected fields.Recommendation: validate/allowlist schema changes in higher envir...
Most organizations don’t have a data problem anymore.They have a data access and usability problem.The dashboards exist. The warehouses are modernized. The lakehouse is running. Yet business teams still wait days for answers because analytics remains...
The Dashboard Era Is Ending. Conversation Is Replacing It.
Enterprise AI becomes difficult to govern as useful projects accumulate. A machine learning team ships a forecasting model. A data engineering team automates pipeline refreshes. Another group connects a generative AI assistant to internal documentati...
I'm building a live cost estimator that doesn't have to wait for the system tables or billing data to update. It gives me immediate cost feedback every second and I'm sharing the development journey on YouTube.I already have live costs estimates for ...
Hi everyone,I recently took a look into a silent cost driver in many data platforms: the default choice between managed and external tables in Unity Catalog.It is very common for teams to default to external tables, but this choice often leads to acc...
Hi all,I have been working with Databricks Asset Bundles (now Declarative Automation Bundles) and kept running into the same friction point: there is no easy way to visually inspect a bundle locally before you deploy.Of course, you can read the YAML,...
Hi @eniwoke !Glad to see that ! I will try it for sure If you need any contribution from my side please let me know !
For years, there was no simple way to disable a single task in a Databricks workflow. Let that sink in If you wanted to skip a task, you had to get creative - Add custom flags - Wrap logic in if/else blocks - Or build your own workaround just to not...
What Is ABAC and Why Does It Matter?Attribute-Based Access Control (ABAC) is a data governance model now available in Databricks, designed to offer fine-grained, dynamic, and scalable access control for data, AI assets, and files managed through Data...
I wanted to check if ABAC (Attribute-Based Access Control) policies can be applied to metric views in Databricks.I have successfully applied ABAC policies on a fact table, and they are working as expected. However, when I query a metric view that use...
DataHacks 2026: University Alliance in Action at UCSD How a single weekend of hands on exposure creates the next generation of Databricks advocates Workshop Lead: Anjana Sriram Why University Alliances Matters in the Field Early in my career, the ...
BI compatibility mode allows us to query Unity Catalog Metric Views from external BI tools. It allows Databricks to rewrite queries generated by tools like Power BI, ensuring complex metric view logic is executed correctly while appearing as standard...
Big shift coming for Azure Databricks usersStarting 30 September 2026, all new Azure Databricks workspaces will be Unity Catalog only.- No DBFS root.- No Hive Metastore.- No legacy runtimes below 13.3 LTS.- No “old way” of doing things.Disabling DBFS...
Hi guys,There was an email message sent by Azure Databricks with this information. So no direct link, just information sent by Databricks to users.
Hi everyone,I recently dealt with a frustrating scenario: a Databricks job that usually takes minutes ran for 18 hours without failing, quietly consuming compute and blocking downstream pipelines.The driver hadn't crashed, and the job hadn't failed—i...
Ever deleted a pipeline… and accidentally wiped out the data with it? Databricks just introduced a beta feature that lets you decouple pipelines from the tables they manage.Lakeflow Spark Declarative Pipelines were desinged with data-as-code approach...
https://medium.com/@bijumathewt/databricks-vector-search-integration-powering-claude-desktop-with-mcp-838c591b50b5
Good Question Edonaire. There 2 options here. Continuous and Triggered. If continuous, it will take care of incrimental load automatically, but costly.Triggered can be filed by setting up job with a task to run the "Refresh" code.
| User | Count |
|---|---|
| 85 | |
| 75 | |
| 70 | |
| 63 | |
| 44 |
