I read about a team demo about their shiny new "autonomous AI agent” leadership. It queried databases, generated reports, sent Slack notifications—all hands-free. Impressive stuff. Two weeks later? $12K in API costs, staging data accidentally pushed ...
We can ingest Excel into Databricks, including natively from SharePoint. It was top news in December, but in fact is part of a big strategy which will allow us to ingest any format from anywhere in databricks. Foundation is already built as there is ...
This use case demonstrates how a SaaS product can be deployed for multiple customers or business units, ensuring data isolation at every layer through workspace separation, fine-grained access control with Unity Catalog, and secure processing using U...
Good breakdown of the Databricks storage and catalog isolation patterns.One thing to keep in mind: workspace binding and Unity Catalog handle data isolation well, but the authentication layer is where tenant context gets established first. Without pr...
Dashboards now offer more flexibility, allowing us to use another field or expression to label or sort the chart. See demo at:- https://www.youtube.com/watch?v=4ngQUkdmD3o&t=893s- https://databrickster.medium.com/databricks-news-week-52-22-december-2...
We’ve recently created a new Excel reader function, and I decided to have a play around with it. I’ve used an open dataset for this tutorial, so you can follow along too. Using file available here - https://www.ons.gov.uk/employmentandlabourmarket/pe...
Nice example!
When something goes wrong, and your pattern involves daily MERGE operations in your jobs, backfill jobs let you reload multiple days in a single execution without writing custom scripts or manually triggering runs. Read more:- https://www.sunnydata.a...
More and more resources are available under DABS. One of the newest additions is the alerts resource.
I just logged in to the community edition for the last time and spun up the cluster for the last time. Today is the last day, but it's still there. Haven't logged in there for a while, as the free edition offers much more, but it is a place where man...
This article provides an overview of key Databricks features and best practices that protect Gold tables from accidental deletion. It also covers the implications if both the Gold and Landing layers are deleted without active retention or backup. Cor...
Thanks for sharing this. Time Travel is applicable all tables in Databricks NOT restricted to gold.
I recently saw a business case in which an external orchestrator accounted for nearly 30% of their total Databricks job costs. That's when it hit me: we're often paying a premium for complexity we don't need. Besides FinOps, I tried to gather all the...
There is a new direct mode in Databricks Asset Bundles: the main difference is that there is no Terraform anymore, and a simple state in JSON. It offers a few significant benefits: - No requirement to download Terraform and terraform-provider-databr...
See the comments below for a runnable notebookThroughout my career I have worked at several companies that handle sensitive data; including PII, PHI, EMR, HIPPA, Class I/II/III FOMC - Internal (FR). One entity I worked at even required a Department O...
We just failed a HIPAA audit, they asked why our pipelines had patients names in the if the pipeline didnt need that info, they recommended it to be encrypted. We thought S3 encryption was good enough.We implemented row level encryption by extending ...
Stream–stream joins are one of the most powerful features in Databricks Structured Streaming – and also one of the easiest to misconfigure. As soon as you move from simple append-only pipelines to real-time correlations across multiple streams (order...
Interesting and very Insightful @Divaker_Soni.
Databricks is a popular unified data analytics platform known for its powerful data processing capabilities and seamless integration with Apache Spark. However, managing and optimizing costs in Databricks can be challenging, especially when it comes ...
@Second Reply You’re right just printing out selected_pool isn’t enough to actually leverage dynamic cluster sizing at runtime. In practice, the value of selected_pool would feed directly into your Databricks cluster creation API or workflow automati...
Meta Ads is now a native data source in DatabricksDatabricks just announced a Meta Ads connector (Beta) powered by Lakeflow Connect, making it easy to ingest advertising data directly into Databricks—no custom APIs, no CSV exports, no brittle scripts...
Have to test it. Confluence Lakeflow Connect is amazing, so I suspect that one also!
| User | Count |
|---|---|
| 85 | |
| 75 | |
| 72 | |
| 60 | |
| 43 |
