Databricks Community

AbhaySingh

I've been working with Unity Catalog's lineage capabilities for a while now, and I have to say—this is what lineage should have always been. Not a separate tool to configure. Not a manual process to maintain. Just automatic, real-time visibility into...

AbhaySingh

How Ontos bridges the gap between technical metadata and business meaning Here's a scenario that might sound familiar. You've got Unity Catalog humming along—tables are registered, lineage is tracked, access controls are in place. Technically, every...

AbhaySingh

We've all been there. You're excited about the lakehouse, you see the value clear as day, and then you try explaining it to a coworker and their eyes glaze over. Slides don't cut it. Documentation links get ignored. What actually works? Showing them...

AbhaySingh

You can now run distributed ML (Spark MLlib in Python, Optuna tuning, MLflow Spark, Joblib Spark) on serverless notebooks/jobs and on standard clusters, not just dedicated ML clusters.It reuses the same Unity Catalog + Lakeguard stack you already use...

AbhaySingh

Last month, our nightly CDC pipeline started timing out. What used to complete in 20 minutes was now crawling past the 4-hour mark—and failing. The culprit? A MERGE statement against a 2.3TB Delta table with 800 million rows that had grown steadily o...

AbhaySingh

There is no public commitment or release forecast for fully automated cross-system monitoring or UI linking system.compute, system.lakeflow, and system.mlflow without custom queries. There is definitely potential for future releases in this area but ...

AbhaySingh

Is following an option for you? https://www.databricks.com/blog/announcing-storage-optimized-endpoints-vector-search Also, good to have your SA address this further as underlying issue may be somewhere else - model serving performance for instance.

AbhaySingh

Root cause in plain English The lookup client is trying to read SQL-style credentials like PREFIX_USER/PREFIX_PASSWORD for a third‑party online store and the “prefix” is empty, so it searches for “_USER” and fails. That auth scheme applies only to th...

AbhaySingh

Support for Spark 4.0 is still an open issuse https://issues.apache.org/jira/browse/HUDI-7915 Please use the preview for testing but you may have to think about design for production with supported releases.

AbhaySingh

Oracle suite of of cloud platforms ( such as cusomer feedback system / CX) typically offer API endpoints . You'd want to use Databricks jobs to fetch from APIs at a cadence per your requirments to medallion layers from go from there - from AI/BI Geni...

Databricks Community

User Stats

User Activity

Unity Catalog Lineage: Lineage That Just Works

Semantic Bridge to Unity Catalog

Your Coworkers Don't Get Databricks? Here's How to Fix That.

Distributed ML on Databricks Serverless

Why Your Delta Lake MERGE Takes Forever (And How to Fix It)

Re: Databricks Systems Tables Link

Re: Vector search index creation is incredibly slow

Re: Databricks Model Serving Endpoint Fails: “_USER not found for feature table”

Re: Create a Hudi table with Databrick 17

Re: What is the best way to connect Oracle CRM cloud from databricks?