As data engineering teams scale out lakehouses and cloud data warehouses, a silent platform killer inevitably creeps in: runaway query costs.In a distributed environment like Databricks, a single unoptimized query whether it’s an accidental Cartesian...
As someone who spends hours every day building notebooks, debugging pipelines, and experimenting with AI agents in Databricks, there was always one thought in the back of my mind.“Why can’t Genie Code have a full-screen workspace where I can manage e...
Managed Tables, External Tables, Foreign Tables and... .. Catalog Managed Tables. If you are databricks user, you need to know why they are the best of all! https://databrickster.medium.com/catalog-commits-make-your-managed-delta-layer-safer-and-more...
This is included with the Databricks managed table; there is no separate catalog managed table.
The rapid evolution of AI agents has transformed how organizations automate tasks, generate insights, and accelerate software development. However, as teams adopt multiple AI models, frameworks, and agent orchestration tools, managing these systems e...
As organizations continue to scale their data platforms, managing consistency, governance, and transactional integrity across multiple datasets has become increasingly challenging. While Delta Lake has long provided robust ACID transactions at the ta...
In this video, I'm going to show you how to automate resource creation in Azure Databricks using the Databricks SDK for Python. If you've ever found yourself manually creating catalogs, schemas, and volumes in the Databricks UI repeatedly, this video...
Databricks Apps run 24/7 with no native scale-to-zero. ~$350/mo per app (720 hrs), but internal dashboards/admin tools only get used a few hours/day. You pay for 100%, use ~15%.We had a test app run idle for 69 days before anyone noticed. ~$800 waste...
Seems there is so much confusion on the upcoming Genie pricing.Databricks Genie pricing, made simple:Free monthly usage stays Only usage beyond it is billed (DBUs)Paid starts July 6, 2026Admins set budgets to cap spendGenie did NOT become paid-only. ...
If you don’t know which permissions (scopes) to add to the access token (since some processes can require many scopes), just set access tokens to auto-scoping, and after 30 days, it will set the required scope based on usage during the first 30 days....
Databricks LakeWatch- Security for the Agentic Era:What is Lakewatch?Lakewatch is a new security product, essentially a comprehensive security camera system for your entire company’s computer systems, powered by AI.Two significant developments occurr...
Databricks Genie vs Databricks Genie Code – What's the Difference?Many people assume Genie and Genie Code are the same, but they are designed for completely different audiences and use cases.GenieWho it’s for: All employees, from Marketing Managers t...
You can now embed a Databricks Genie Space directly into an external website or application.What does this mean?Instead of asking your business users to log into Databricks every time they want to query data, you can bring Genie to them right inside ...
If you deploy a bundle from the UI in development mode (source-linked deployment) and edit it in the UI, changes are now propagated to the files. Really useful for jobs and dashboards. For jobs, of course, please always review in GIT as results in so...
Catalog Commits are here, and they are bringing to databricks and UC: - concurrency control, because Unity Catalog coordinates the winning commit- governance, because supported clients resolve table state through Unity Catalog- lays the foundation fo...
Lakebase can scale up more, and Apps are now getting horizontal scaling. Seems like #databricks is the best place to run your app now, any app. https://databrickster.medium.com/databricks-news-cli-v-1-0-0-ai-tools-last-updated-25th-may-767ef39abe8a
