cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Khasim_1
by • Contributor
  • 754 Views
  • 2 replies
  • 3 kudos

Resolved! Best practices for SLA monitoring and automated retries across hundreds of Lakeflow Jobs

Hi everyone,We are in the process of migrating our legacy Airflow DAGs (largely time-based cron schedules) to Lakeflow Jobs, and I want to fully leverage the platform's event-driven capabilities rather than just replicating the old "timer-based" patt...

  • 754 Views
  • 2 replies
  • 3 kudos
Latest Reply
Islam_hoti
Contributor III
  • 3 kudos

Hi,Migrating the same way. The messy parts.On file arrival triggers, first check whether your external locations have file events enabled, because the behaviour differs sharply. Without them you get a cap of 50 triggered jobs per workspace and at mos...

  • 3 kudos
1 More Replies
Khasim_1
by • Contributor
  • 1228 Views
  • 4 replies
  • 5 kudos

Resolved! Non-deterministic ROW_NUMBER() results across Unity Catalog environments

We have a notebook that calculates the "mode" (most frequent value) of a customer_parent_name field using ROW_NUMBER() OVER (PARTITION BY ... ORDER BY count DESC). The exact same notebook code is deployed to two Unity Catalog-governed environments (D...

  • 1228 Views
  • 4 replies
  • 5 kudos
Latest Reply
Islam_hoti
Contributor III
  • 5 kudos

Hi,Your diagnosis is correct, and the fix is simpler than you are expecting.On question one, yes, this is expected and it is not a Databricks or Unity Catalog behaviour. ROW_NUMBER with an ORDER BY that does not define a total order is non determinis...

  • 5 kudos
3 More Replies
Khasim_1
by • Contributor
  • 691 Views
  • 2 replies
  • 3 kudos

Resolved! igrating from Cron-based Airflow to Lakeflow's Data-Aware Triggers — Real-world experiences?

Hi everyone,We are in the process of migrating our legacy Airflow DAGs (largely time-based cron schedules) to Lakeflow Jobs, and I want to fully leverage the platform's event-driven capabilities rather than just replicating the old "timer-based" patt...

  • 691 Views
  • 2 replies
  • 3 kudos
Latest Reply
Islam_hoti
Contributor III
  • 3 kudos

Hi,Migrating the same way. The messy parts.On file arrival triggers, first check whether your external locations have file events enabled, because the behaviour differs sharply. Without them you get a cap of 50 triggered jobs per workspace and at mos...

  • 3 kudos
1 More Replies
jeremy98
by • Honored Contributor
  • 14061 Views
  • 12 replies
  • 4 kudos

Resolved! ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'

Hello community,I installed databricks extension on my vscode ide. How to fix this error? I created the environment to run locally my notebooks and selected the available remote cluster to execute my notebook, what else?I Have this error: ImportError...

  • 14061 Views
  • 12 replies
  • 4 kudos
Latest Reply
jeremy98
Honored Contributor
  • 4 kudos

@unj1m yes, as Alberto said you don't need to install pyspark, it is included in your cluster configuration.

  • 4 kudos
11 More Replies
Hamü123
by • New Contributor II
  • 578 Views
  • 2 replies
  • 2 kudos

Choosing between a materialized view and a job-maintained Delta table for an expensive SQL query

I am relatively new to Databricks and do not yet have much experience with materialized views, Delta tables, incremental processing, or the computational costs associated with refreshing persisted results.I currently have a Databricks SQL view that c...

  • 578 Views
  • 2 replies
  • 2 kudos
Latest Reply
data_pulse
Contributor
  • 2 kudos

Adding to the above, can do one small validation before deciding between MV and a custom Delta/MERGE solution.Databricks now lets you check whether the MV query is structurally eligible for incremental refresh:EXPLAIN CREATE MATERIALIZED VIEW mv_orde...

  • 2 kudos
1 More Replies
hassanashas
by • New Contributor II
  • 621 Views
  • 2 replies
  • 0 kudos

Are attached-but-idle serverless notebooks billed? Idle timeout? 32 min job, 86 min billed

Hi all, hope everyone is doing well. Two questions I am hoping someone can help with: is there a way to set an idle timeout for serverless notebooks, and is an attached-but-idle serverless notebook billed while it sits there?Some background on why I ...

hassanashas_0-1789429926568.png hassanashas_1-1789429984131.png
  • 621 Views
  • 2 replies
  • 0 kudos
Latest Reply
data_pulse
Contributor
  • 0 kudos

@Louis_Frolio Thanks, this clarifies the idle billing behaviour.I also observed a very similar pattern in another Azure workspace using the EU West serverless all-purpose SKU. The last visible notebook execution was around 09:56 UTC, while system.bil...

  • 0 kudos
1 More Replies
rachelnguyen
by • New Contributor
  • 671 Views
  • 2 replies
  • 0 kudos

Getting started with Delta Lake basics

Hi all — I am working through the Delta Lake intro and wanted to share a short note on using OPTIMIZE after merge. Happy to hear better patterns.

  • 671 Views
  • 2 replies
  • 0 kudos
Latest Reply
balajij8
Esteemed Contributor III
  • 0 kudos

@rachelnguyen MERGE operation is inherently expensive as it searches the entire target table to find matches, then shuffles data multiple times to write updates/inserts. It generates many small files - especially on partitioned tables where each shuf...

  • 0 kudos
1 More Replies
techinlayers
by • New Contributor III
  • 444 Views
  • 0 replies
  • 1 kudos

Databricks architecture drawn step by step: control plane, compute plane, Delta, Unity Catalog

I kept getting the same interview question, "explain the Databricks architecture", and most answers are a wall of text. So I drew it.The crisp answer: Databricks manages the control plane (web app, job scheduler, notebook metadata). Your compute runs...

  • 444 Views
  • 0 replies
  • 1 kudos
Islam_hoti
by • Contributor III
  • 964 Views
  • 2 replies
  • 2 kudos

Resolved! ConcurrentAppendException on parallel MERGE into the same Delta table, despite disjoint keys

Hi everyone, Hoping someone has been through this, because I am running out of ideas. Setup. DBR 15.4 LTS, Unity Catalog managed Delta table, roughly 900 million rows. Six upstream sources each write into the same target table through their own job, ...

  • 964 Views
  • 2 replies
  • 2 kudos
Latest Reply
Khasim_1
Contributor
  • 2 kudos

Hi @Islam_hoti This is a classic "Optimistic Concurrency Control" bottleneck in Delta Lake. Even with partitions, the MERGE operation requires a metadata-level lock on the table's transaction log, which is where your contention is happening.Here is a...

  • 2 kudos
1 More Replies
carl_hsg
by • New Contributor III
  • 722 Views
  • 2 replies
  • 4 kudos

Resolved! AUTO CDC FROM SNAPSHOT schema evolution in history tracking columns

Does the create_auto_cdc_from_snapshot_flow() allow for expansion of included columns in track_history_column_list or track_history_except_column_list without doing a full refresh? My intended flow is: Bronze - Managed ingestion of table (scd 1) usin...

  • 722 Views
  • 2 replies
  • 4 kudos
Latest Reply
ivanvyd
Contributor
  • 4 kudos

Adding a column and changing the history-tracking configuration need separate checks. Databricks describes adding columns as generally safe without a full refresh:"Plan your schema to avoid changes that require a full refresh. Adding columns is gener...

  • 4 kudos
1 More Replies
km1837
by • Databricks Partner
  • 884 Views
  • 4 replies
  • 0 kudos

AI/BI Dashboards

Hi AllMy client is looking to create a few dashboards in DataBricks instead of Power BI. We already have a template in Power BI (header for each page, information about out BI Page) in each of our power bi report which has colors, fonts, images etc.H...

  • 884 Views
  • 4 replies
  • 0 kudos
Latest Reply
Ashwin_DSA
Databricks Employee
  • 0 kudos

Hi @km1837, Databricks AI/BI Dashboards have solid theming and organisational capabilities that should cover most of what you are used to in Power BI. AI/BI Dashboards don't have a "template file" concept the way Power BI does with .pbit files, but y...

  • 0 kudos
3 More Replies
data_pulse
by • Contributor
  • 1151 Views
  • 4 replies
  • 1 kudos

Resolved! Auto Loader stream fails on RocksDB checkpoint after enabling managed file events

Hello Community.We recently enabled the following option on several existing Auto Loader streams:"cloudFiles.useManagedFileEvents": "true"Most streams continued working, but one now fails while restoring its existing checkpoint, referencing a file si...

  • 1151 Views
  • 4 replies
  • 1 kudos
Latest Reply
data_pulse
Contributor
  • 1 kudos

After further investigation into it, have understood the root cause and the fix for it.After enabling Auto Loader managed file events across the platform: .option("cloudFiles.useManagedFileEvents", "true")the failed stream started discovering RocksDB...

  • 1 kudos
3 More Replies
Sanjeeb2024
by • Valued Contributor II
  • 1248 Views
  • 3 replies
  • 2 kudos

Resolved! Databricks Apps - Deployment process - Need help !!

Hi all,Do we have any guidelines or a CI/CD approach for deploying a Databricks app from one environment (e.g., DEV) to PROD?We are currently facing the following challenge:We created a Databricks app in the DEV environment through the UI. At runtime...

  • 1248 Views
  • 3 replies
  • 2 kudos
Latest Reply
Sanjeeb2024
Valued Contributor II
  • 2 kudos

Thank you for the details. Yes, I agree we should adopt the DABs for the deployment and define the resources while creating the app ( which can be consider as as solution). However if the permission module is managed separately, lets say via terrafor...

  • 2 kudos
2 More Replies
Islam_hoti
by • Contributor III
  • 1254 Views
  • 2 replies
  • 3 kudos

Resolved! Best Practice for Handling Schema Evolution with Auto Loader in Production?

Hi Databricks Community,I’d like to hear how other Data Engineers are handling schema evolution with Auto Loader in production environments.Consider the following scenario:We have a continuously running ingestion pipeline using Auto Loader that proce...

  • 1254 Views
  • 2 replies
  • 3 kudos
Latest Reply
J_Chakraborty
New Contributor II
  • 3 kudos

You can absolutely leverage Databricks Genie and the new Genie Ontology to handle downstream changes caused by addNewColumnsWithTypeWidening, and handle the changes using Genie ontology, but it requires understanding the division of labor between you...

  • 3 kudos
1 More Replies
Labels