Databricks Community

SumeshKashyap

Hi everyone! I built a near-real-time app that follows every tram and metro train in Helsinki using only Databricks: no Kafka, no extra servers, and the whole thing deploys from one repository.What the app doesA live map of every tram and metro train...

SumeshKashyap

Streaming tables are append-only by design, so they can't overwrite. Use a materialized view that keeps only the rows from the newest file, via the _metadata column: sqlCREATE OR REFRESH MATERIALIZED VIEW orders ASSELECT * EXCEPT (file_time)FROM ( S...

SumeshKashyap

These are great questions. The third one in particular is where most setups I've seen get stuck, so thanks for starting this thread. Here's what tends to work, taking your questions one at a time.Q1: Are you computing cost per task from traces and to...

SumeshKashyap

Based on the error message shared, there are two issues in the corrected rule, plus one in the logic. Why sql_query is failing: - condition_column must be a BOOLEAN column. DQX treats it as "TRUE = this row fails the check" and wraps it in a CASE WHE...

SumeshKashyap

The pattern that keeps this manageable at scale is treating recipients as code and running rotation as a scheduled job. Here's the breakdown: - Set a token lifetime at the metastore level. Open-sharing bearer tokens expire according to the metastore'...

SumeshKashyap

This one's really a network/download-client issue more than a Databricks one, but since the end goal is presumably getting the data into Databricks, a couple of angles: - Browser downloads are single-connection and easy to bottleneck -- 8-9 Mbps on a...

Databricks Community

User Stats

User Activity

Built a live Helsinki tram tracker on Databricks with Lakeflow, Lakebase, Genie and Apps

Re: Databricks SDP (Spark declarative Pipelins) Overwrite table

Re: How are you attributing agent cost per successful task, not just per token?

Re: Issue in DQX DQGeneration with AI - DQrules generated with sql_query function having syntax erro

Re: Scaling Delta Sharing across Non-Databricks Recipients: Handling Credential Rotation and Lifecyc

Re: Optimizing B2B E-Commerce & Construction Supply Chain Pipelines (Data Architecture Case Stud