Databricks Community

Islam_hoti

You cluster a table on a column, run the query, and nothing improves. The layout command completed successfully. The column is definitely in the filter. The file count in the query profile has not moved.Before assuming the clustering did not work, ch...

Islam_hoti

Hi everyone,A question I keep having with my team and I would like to hear how others think about it.A lot of the jobs we run are not big. Plenty of our pipelines process a few gigabytes, some considerably less. We run them on Spark because that is w...

Islam_hoti

Hi everyone,Trying to work out whether this is expected or whether I have misconfigured something.We enabled Photon on a job cluster running a nightly aggregation over roughly 2TB. The expectation was the usual improvement. What we got instead was ru...

Islam_hoti

Hi everyone,Looking for the right pattern here rather than a workaround.Setup. DBR 15.4 LTS, Unity Catalog. A silver streaming table reads from a bronze Delta table with a normal streaming read. Bronze is append only in the ordinary course of busines...

Islam_hoti

Hi all,We have a Silver Delta table (~5 TB, growing ~20 GB/day) that is updated every hour with a MERGE on order_id. Right now it's partitioned by order_date. Most MERGE batches touch the last 3 days, but late updates can reach orders up to 90 days o...

Islam_hoti

Hi,Great question. This is where most Medallion implementations either scale or get messy. Here's the pattern that has worked well, based on Kimball conformed dimensions mapped onto Unity Catalog.Quick note on naming: in Databricks, "Zerobus" is a pr...

Re: Databricks AI/BI Get Session User

Hi,Short answer: No. There's no built-in function or parameter that returns the viewer's identity while a dashboard runs on the publisher's credentials.Why: When you publish with Share data permissions (the default, formerly called "Embed credentials...

Islam_hoti

Hi,The basic rule: the bundle manages secret scopes and references to secrets, never the secret values.1. Keeping secrets out of databricks.yml: Only reference scope and key names. Jobs resolve the secret at runtime:In code: dbutils.secrets.get(scope...

Islam_hoti

Hi,In practice most teams end up with a hybrid: keep raw PII in one tightly restricted place, and use dynamic masking everywhere else.1. Does redacting at Silver (A) create friction? Yes, often. Once PII is hashed or dropped in Silver, the raw data i...

Islam_hoti

Hi,1. Planning time vs. views: Planning overhead is usually small. Row filters and column masks are inlined into the plan much like a view, and identity functions such as is_account_group_member() are resolved once per query, not per row. On 100M+ ro...

Databricks Community

User Stats

User Activity

The 32nd Column: Why Your Z-Order or Clustering Key May Be Doing Nothing

At what data size do you stop reaching for Spark?

Photon enabled but a large share of the plan is falling back, cost up and runtime flat

Streaming read fails with "Detected a data update" after a restatement job touches the source

Liquid clustering vs. partitioning for a 5 TB Silver table with frequent MERGEs

Re: Implementing a "Zero-Bus" Architecture in a Lakehouse: Best Practices for Shared Dimen

Re: Databricks AI/BI Get Session User

Re: Managing "Secret" Injection in DABs across Dev/Stage/Prod

Re: Architectural Pattern for PII: "Redact at Ingestion" vs. "Dynamic Masking at Cons

Re: Managing Row-Level Security (RLS) vs. Views: Performance impacts at scale