If you are new to Databricks and want to get started, I have started a playlist. 26 videos already published, more in the pipeline. Happy learning !!Linkedin post: https://www.linkedin.com/posts/sudarshan-koirala_databricks-dataengineering-datascienc...
Declarative pipelines are among the best ways to deduplicate your data, especially for dimensions. From AUTO_CDC() to advanced deduplication quality check #databricks https://databrickster.medium.com/deduplicating-data-on-the-databricks-lakehouse-5-w...
Published a new blog detailing how I used Codex to configure Databricks AI Dev Kit on my local Mac and then implemented a simple tool-calling agent on Databricks in a step-by-step workflow.This post is part of my ongoing series focused on practical d...
Codex, Claude, Gemini blocked? No problem. Route everything through Databricks AI Gateway. #databricks https://databrickster.medium.com/databricks-news-2026-week-8-16-february-2026-to-22-february-2026-f2ec48bc234f
More under DABS! External locations are now available as DABS code. I hope that credentials will be available soon, too, so it will be possible to reference the credential resource from an external location. #databricks https://medium.com/@databricks...
This is great, so eventually DAB might help replacing Terraform overhead for these platform resources or any other infrastructure maintenance and governance.
Many teams think migrating from Azure Data Factory to Databricks Lakeflow Jobs is just a pipeline rewrite. It’s not.It’s about:* Simplifying over-engineered architectures* Reducing orchestration and monitoring overhead* Aligning data engineering with...
Catalogs are now under DABS, and I am happy to say goodbye to Terraform and to manage all UC grants in DABS. #databricks https://databrickster.medium.com/databricks-news-2026-week-8-16-february-2026-to-22-february-2026-f2ec48bc234f
Hi everyone, I wanted to share a quick technical tip for those looking to optimize their SQL Warehouse configurations, especially when dealing with intermittent ad hoc queries or AI BI Dashboards. The Challenge: When setting up a SQL Warehouse, the D...
It is possible to tag queries. That functionality is also supported by external clients (Jdbc/dbt/Power BI, etc.) #databricks https://databrickster.medium.com/databricks-news-2026-week-8-16-february-2026-to-22-february-2026-f2ec48bc234f
Do you know that instead of SELECT * FROM TABLE, you can just use TABLE? TABLE is just part of pipe syntax, so you can always add another part after the pipe. Thanks to Martin Debus for noticing the possibility of using just TABLE. #databricks https:...
I am back, and runtime 18.1 is here, and with it INSERT WITH SCHEMA EVOLUTION https://databrickster.medium.com/databricks-news-2026-week-8-16-february-2026-to-22-february-2026-f2ec48bc234f
For more than 50 years, software engineering has been guided by principles like The Mythical Man-Month, Brooks’s Law, Hofstadter’s Law, Parkinson’s Law, and the Iron Triangle.But those laws were built around a specific reality: human cognitive limits...
Love this post, Sudhir.It's been fascinating to see how quickly the principles in TMMM are evolving in the last 6 months, with the rise of agentic AI. It's a shame Fred Brooks is not around to see this technology shift.
Agentic quality monitoring is available in databricks. But tooling alone is not enough. You need a clearly defined Data Quality Pillar across your Lakehouse architecture. #databricks https://www.sunnydata.ai/blog/databricks-data-quality-pillar-ai-rea...
At the Lakehouse, we don't enforce Primary Keys, which is why the deduplication strategy is so important. One of my favourites is using transformWithStateInPandas. Of course, it only makes sense in certain scenarios. See all five major strategies on ...
When a new external location is created, file events are created by default. The autoloader from runtime 18.1 defaults to file notification mode.It is just a few weeks after TRIGGER ON UPDATE was introduced. more news:https://databrickster.medium.com...
