We've all been there. You're excited about the lakehouse, you see the value clear as day, and then you try explaining it to a coworker and their eyes glaze over. Slides don't cut it. Documentation links get ignored. What actually works? Showing them...
In today’s data-driven world, organisations are drowning in information. From customer transactions and IoT sensor readings to social media interactions and operational logs, the volume and variety of data continue to grow exponentially. Yet many org...
Thank you, @Louis_Frolio ! My next post is about Data Governance with Unity Catalog, stay tuned!!
Disaster recovery is possible in Unity catalog now?Means, for data level, we have enabled with geo redundancy, what about the objects, permissions, an other components in Unity catalog ? Can we restore the unity catalog metadata in another region ?
Official product release in development will be available as PrPr in a few months.
If you are going with DABS into a production environment, a CLI version is considered best practice. Of course, you need to remember to bump it up from time to time. Learn more: - https://databrickster.medium.com/managing-databricks-cli-versions-i...
You can now run distributed ML (Spark MLlib in Python, Optuna tuning, MLflow Spark, Joblib Spark) on serverless notebooks/jobs and on standard clusters, not just dedicated ML clusters.It reuses the same Unity Catalog + Lakeguard stack you already use...
Last month, our nightly CDC pipeline started timing out. What used to complete in 20 minutes was now crawling past the 4-hour mark—and failing. The culprit? A MERGE statement against a 2.3TB Delta table with 800 million rows that had grown steadily o...
Check this medium article https://medium.com/@bijumathewt/real-time-sql-server-to-databricks-pipeline-using-debezium-kafka-and-delta-lake-26c3e191ce51?postPublishedType=initial
would you not be able to use Lakeflow Managed connector for SQL Server in this case?
Tired of complex governance workflows? We’ve successfully combined the power of multi-agent AI with state-of-the-art orchestration tools to automate Databricks Unity Catalog (UC) access management!This isn’t just a basic script — it’s an intelligent,...
https://medium.com/@bijumathewt/real-world-langgraph-crewai-application-intelligent-databricks-alert-system-with-ai-7ba36e23f7a2You will love above article
You will love this article. Watch it. https://medium.com/@bijumathewt/real-world-langgraph-crewai-application-intelligent-databricks-alert-system-with-ai-7ba36e23f7a2
Delta Lake 4.0 is the next major open-source release aligned with Spark 4.x, adding first-class Variant for semi-structured data, safer Type Widening, improved DROP FEATURE, better transaction log handling, and a new multi-engine story via Delta Kern...
Databricks just solved a huge problem - unlocking the value from unstructured data. One of the biggest challenges enterprises face when scaling agents is access to unstructured data. Nearly 80% of enterprise knowledge is trapped in PDFs, reports, and...
Thanks for sharing, @venkat-raghavan! ai_parse_document turns documents into governed, queryable assets, it's definitely the puzzle piece all needed.
This is super amazing and super powerful -https://docs.databricks.com/aws/en/generative-ai/mcp/managed-mcpEssentially we you can use Databricks managed MCP servers to seamlessly integrate Databricks features into any AI Agent. That's the multiplier ...
If you’ve ever hacked together a one-off script to pull data from some random API into Spark, you’re exactly who the new Python Data Source API is for. Databricks has made this API generally available on Apache Spark™ 4.0 with Databricks Runtime 1...
I have seen multiple glue jobs pulling data from such systems. This is certainly a solution to simplify and bring governance to it. will look forward to implement it.#Apache-4
You spend hours sizing clusters, tuning autoscaling configurations, and optimizing instance types.Clusters sit idle burning money between jobs or take minutes to start when you need them immediately.Managing compute infrastructure becomes a full-time...
| User | Count |
|---|---|
| 85 | |
| 76 | |
| 73 | |
| 60 | |
| 43 |
