cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

zenwanderer
by • New Contributor II
  • 1979 Views
  • 5 replies
  • 0 kudos

Kill/Cancel a Notebook Cell Running Too Long on an All-purpose Cluster

Hi everyone, I’m facing an issue when running a notebook on a Databricks All-purpose cluster. Some of my cells/pipelines run for a very long time, and I want to automatically cancel/kill them when they exceed a certain time limit.I tried setting spar...

  • 1979 Views
  • 5 replies
  • 0 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 0 kudos

Greetings @zenwanderer, I did some digging and here is what I found. Your follow-up narrows the ask: you want the workspace, not each developer, to catch and kill any interactive cell that runs past a threshold. Short answer: there's no supported, bu...

  • 0 kudos
4 More Replies
Pavel_Sinkevich
by • New Contributor III
  • 1177 Views
  • 6 replies
  • 8 kudos

CSV export from AI/BI Dashboard table widget does not preserve visual column order

SummaryWhen downloading data as CSV from a table widget on an AI/BI Dashboard (kebab menu → Download → Data → Download CSV), the column order in the exported file does not match the column order displayed in the dashboard. All aggregated (measure) co...

  • 1177 Views
  • 6 replies
  • 8 kudos
Latest Reply
Akash_Murade
New Contributor III
  • 8 kudos

This issue occurs if your dashboard has a semantic model attached i.e., you've clicked "Get started with relationships" even if it doesn't have any relationships defined. Just having the semantic model present is enough to trigger it.The problem is i...

  • 8 kudos
5 More Replies
Khasim_1
by • Contributor
  • 960 Views
  • 2 replies
  • 3 kudos

Resolved! Managing "Secret" Injection in DABs across Dev/Stage/Prod

Hi everyone, I am fully moving our team to Databricks Asset Bundles (DABs) for CI/CD, but I’m struggling with the "Secret" management pattern.How are you handling the injection of secrets (like API keys for ingestion) into DABs without hardcoding any...

  • 960 Views
  • 2 replies
  • 3 kudos
Latest Reply
Islam_hoti
Contributor III
  • 3 kudos

Hi,The basic rule: the bundle manages secret scopes and references to secrets, never the secret values.1. Keeping secrets out of databricks.yml: Only reference scope and key names. Jobs resolve the secret at runtime:In code: dbutils.secrets.get(scope...

  • 3 kudos
1 More Replies
Khasim_1
by • Contributor
  • 1048 Views
  • 3 replies
  • 6 kudos

Resolved! Architectural Pattern for PII: "Redact at Ingestion" vs. "Dynamic Masking at Consumption

 Hi everyone, we are defining our PII strategy in Unity Catalog. We are split on whether to: A) Redact/Hash PII at the Silver layer (permanent change), or B) Keep PII in Silver and use Dynamic Data Masking at the Gold/View layer.Does "Redact at Inges...

  • 1048 Views
  • 3 replies
  • 6 kudos
Latest Reply
Islam_hoti
Contributor III
  • 6 kudos

Hi,In practice most teams end up with a hybrid: keep raw PII in one tightly restricted place, and use dynamic masking everywhere else.1. Does redacting at Silver (A) create friction? Yes, often. Once PII is hashed or dropped in Silver, the raw data i...

  • 6 kudos
2 More Replies
Khasim_1
by • Contributor
  • 1207 Views
  • 3 replies
  • 4 kudos

Resolved! Managing Row-Level Security (RLS) vs. Views: Performance impacts at scale

 Hi everyone, we are designing our security layer in Unity Catalog. We are debating between using Row-Level Security (RLS) predicates versus standard SQL Views for masking PII data.When applying complex RLS predicates to tables with >100M rows, have ...

  • 1207 Views
  • 3 replies
  • 4 kudos
Latest Reply
Islam_hoti
Contributor III
  • 4 kudos

Hi,1. Planning time vs. views: Planning overhead is usually small. Row filters and column masks are inlined into the plan much like a view, and identity functions such as is_account_group_member() are resolved once per query, not per row. On 100M+ ro...

  • 4 kudos
2 More Replies
Khasim_1
by • Contributor
  • 713 Views
  • 2 replies
  • 3 kudos

Resolved! Scaling Delta Sharing across Non-Databricks Recipients: Handling Credential Rotation and Lifecycle

Hi everyone,Hi everyone, I’m architecting a data-sharing program using Delta Sharing for external partners who do not have a Databricks workspace.How are others automating the recipient credential lifecycle? Do you have an automated process for rotat...

  • 713 Views
  • 2 replies
  • 3 kudos
Latest Reply
Islam_hoti
Contributor III
  • 3 kudos

Hi,1. Credential lifecycle: If your partners have their own identity provider (Entra, Okta, etc.), consider OIDC token federation instead of bearer tokens. The partner's IdP issues short-lived tokens, so there's nothing for you to rotate or leak. It ...

  • 3 kudos
1 More Replies
Khasim_1
by • Contributor
  • 924 Views
  • 3 replies
  • 6 kudos

Resolved! Best practice to enforcing row-level security across multiple catalogs sharing same schema structure

Hi Everyone,We have identical schema structures replicated across three catalogs (dev, qa, prod) to support environment isolation. We now need to apply row-level security (e.g., restricting sales reps to only see their own region's data) consistently...

  • 924 Views
  • 3 replies
  • 6 kudos
Latest Reply
Islam_hoti
Contributor III
  • 6 kudos

Hi,1. Sharing functions across catalogs: You don't need three copies. Row filters and ABAC policies take a fully qualified function name, so you can keep one UDF in a central catalog (e.g. governance.security.region_filter). When a cross-catalog refe...

  • 6 kudos
2 More Replies
Ericsson
by • New Contributor II
  • 8778 Views
  • 6 replies
  • 1 kudos

SQL week format issue its not showing result as 01(ww)

Hi Folks,I've requirement to show the week number as ww format. Please see the below codeselect weekofyear(date_add(to_date(current_date, 'yyyyMMdd'), +35)). also plz refre the screen shot for result.

result
  • 8778 Views
  • 6 replies
  • 1 kudos
Latest Reply
Rodgers
New Contributor II
  • 1 kudos

Data engineering is an increasingly important field, and communities focused on best practices and practical solutions can be valuable for professionals at every level. Discussions about architecture, optimization, data pipelines, and efficient workf...

  • 1 kudos
5 More Replies
ChristianRRL
by • Honored Contributor II
  • 2145 Views
  • 4 replies
  • 5 kudos

Resolved! Unity Catalog - How to read prod data in dev with appropriate read-only access?

Hi there,Our team is currently migrating to using Unity Catalog. We have two databricks workspaces for dev & prod, and one thing that I'm wondering is if there is a simple/appropriate way to have only two catalogs dev & prod, where the prod databrick...

  • 2145 Views
  • 4 replies
  • 5 kudos
Latest Reply
Nagendra_B
New Contributor II
  • 5 kudos

Yes, we can share prod data into a dev workspace in Unity Catalog with read-only access. Databricks has built this to be remarkably simple without lengthy processes.The only prerequisites are that both workspaces must be in the same region and attach...

  • 5 kudos
3 More Replies
LiresaFerizaj
by • Contributor
  • 702 Views
  • 2 replies
  • 4 kudos

Databricks now deletes unused service principal secrets after 90 days

Here's a small change from this month's release notes that's easy to miss. Databricks now automatically deletes a service principal's OAuth client secret after 90 days without use. This matches how unused access tokens were already cleaned up.Why thi...

  • 702 Views
  • 2 replies
  • 4 kudos
Latest Reply
Khasim_1
Contributor
  • 4 kudos

Great write-up, Liresa — this is a really important operational gotcha that's easy to overlook until it bites during a quarterly job or DR drill. A few thoughts to add to the discussion:Automate tracking instead of manual lists Set up a scheduled che...

  • 4 kudos
1 More Replies
LiresaFerizaj
by • Contributor
  • 576 Views
  • 1 replies
  • 5 kudos

Genie One Foundations: What Data Teams Should Get Right Before Rolling It Out

Hi everyone Genie One has been getting a lot of attention since DAIS, and most of the discussion is about what business users can do with it. I want to focus on the other side: what data teams need in place so it gives trustworthy answers.Quick recap...

  • 576 Views
  • 1 replies
  • 5 kudos
Latest Reply
ivanvyd
Contributor
  • 5 kudos

Thanks, @LiresaFerizaj, appreciate your efforts! I'd add one step to the weekly review: turn recurring mistakes into regression tests, then rerun them whenever metric definitions, table descriptions or agent instructions change.For a Genie Agent, the...

  • 5 kudos
Khasim_1
by • Contributor
  • 979 Views
  • 4 replies
  • 7 kudos

Operationalizing Lakeflow Connect:Handling Upstream Schema Evolution & Historical Backfill

Hi everyone,I’m currently evaluating Lakeflow Connect for our ingestion layer. While the setup for replicating source databases into the Lakehouse is remarkably streamlined, I’m looking for operational best practices from those of you using it in pro...

  • 979 Views
  • 4 replies
  • 7 kudos
Latest Reply
stephen4
Contributor
  • 7 kudos

The point about keeping Silver on an explicit schema contract is really useful, especially during full refreshes. I’d also keep an eye on CDC retention during large backfills so nothing gets missed. 

  • 7 kudos
3 More Replies
AdamLucas
by • New Contributor II
  • 686 Views
  • 3 replies
  • 3 kudos

Which Topics Deserve More Attention Before the Databricks Data Engineer Associate Certification Exam

As soon as I began my preparation for the Databricks Data Engineer Associate Certification exam, I came to the realisation that merely going through the syllabus won’t be sufficient. There are certain topics which are recurrent in practical situation...

  • 686 Views
  • 3 replies
  • 3 kudos
Latest Reply
josepe
New Contributor II
  • 3 kudos

Hey @AdamLucas I agree that just reading the syllabus isn’t enough for the Databricks Data Engineer Associate exam. I focused more on Delta Lake, Spark SQL, data transformation, and workflows because these topics needed a better understanding of how ...

  • 3 kudos
2 More Replies
lessalucas
by • New Contributor II
  • 385 Views
  • 1 replies
  • 1 kudos

New Databricks SQL window fuction: counter_diff

This function was designed to work with cumulative counters in time series. It automatically calculates the difference between the current and previous value, turning a cumulative counter into a per-period variation.Let's look at a practical example....

  • 385 Views
  • 1 replies
  • 1 kudos
Latest Reply
Khasim_1
Contributor
  • 1 kudos

The counter_diff function is a significant upgrade over the traditional LAG() approach for IoT and telemetry data because it is "reset-aware."While a standard subtraction (Current - LAG(Previous)) produces a large negative number when a hardware coun...

  • 1 kudos
nafikazi
by • Contributor
  • 1121 Views
  • 6 replies
  • 3 kudos

Resolved! Power BI Dataflow Gen1 → Databricks Pipeline: Is There a Migration Accelerator or Converter?

Hello Great Analytics professionals:Many organizations have legacy Power BI Dataflow Gen1 solutions and are now considering moving their ETL workloads to Databricks.Is there an existing Databricks migration accelerator, converter, or framework that c...

  • 1121 Views
  • 6 replies
  • 3 kudos
Latest Reply
data_pulse
Contributor
  • 3 kudos

@nafikazi There isn't a one-click Power BI Dataflow Gen1 / Power Query M to Lakeflow conversion path yet available.For a POC, decompose each flow and use Genie Code to accelerate the translation into Databricks SQL/ PySpark.Provide the M query, sourc...

  • 3 kudos
5 More Replies
Labels