Databricks Community

AbhaySingh

We've all been there. You're excited about the lakehouse, you see the value clear as day, and then you try explaining it to a coworker and their eyes glaze over. Slides don't cut it. Documentation links get ignored. What actually works? Showing them...

AbhaySingh

You can now run distributed ML (Spark MLlib in Python, Optuna tuning, MLflow Spark, Joblib Spark) on serverless notebooks/jobs and on standard clusters, not just dedicated ML clusters.It reuses the same Unity Catalog + Lakeguard stack you already use...

AbhaySingh

Last month, our nightly CDC pipeline started timing out. What used to complete in 20 minutes was now crawling past the 4-hour mark—and failing. The culprit? A MERGE statement against a 2.3TB Delta table with 800 million rows that had grown steadily o...

AbhaySingh

Delta Lake 4.0 is the next major open-source release aligned with Spark 4.x, adding first-class Variant for semi-structured data, safer Type Widening, improved DROP FEATURE, better transaction log handling, and a new multi-engine story via Delta Kern...

AbhaySingh

If you’ve ever hacked together a one-off script to pull data from some random API into Spark, you’re exactly who the new Python Data Source API is for. Databricks has made this API generally available on Apache Spark™ 4.0 with Databricks Runtime 1...

AbhaySingh

There is no public commitment or release forecast for fully automated cross-system monitoring or UI linking system.compute, system.lakeflow, and system.mlflow without custom queries. There is definitely potential for future releases in this area but ...

AbhaySingh

Is following an option for you? https://www.databricks.com/blog/announcing-storage-optimized-endpoints-vector-search Also, good to have your SA address this further as underlying issue may be somewhere else - model serving performance for instance.

AbhaySingh

Root cause in plain English The lookup client is trying to read SQL-style credentials like PREFIX_USER/PREFIX_PASSWORD for a third‑party online store and the “prefix” is empty, so it searches for “_USER” and fails. That auth scheme applies only to th...

AbhaySingh

Support for Spark 4.0 is still an open issuse https://issues.apache.org/jira/browse/HUDI-7915 Please use the preview for testing but you may have to think about design for production with supported releases.

AbhaySingh

Oracle suite of of cloud platforms ( such as cusomer feedback system / CX) typically offer API endpoints . You'd want to use Databricks jobs to fetch from APIs at a cadence per your requirments to medallion layers from go from there - from AI/BI Geni...

Databricks Community

User Stats

User Activity

Your Coworkers Don't Get Databricks? Here's How to Fix That.

Distributed ML on Databricks Serverless

Why Your Delta Lake MERGE Takes Forever (And How to Fix It)

Delta Lake 4.0 in the Real World

Python Data Source API

Re: Databricks Systems Tables Link

Re: Vector search index creation is incredibly slow

Re: Databricks Model Serving Endpoint Fails: “_USER not found for feature table”

Re: Create a Hudi table with Databrick 17

Re: What is the best way to connect Oracle CRM cloud from databricks?