- 1050 Views
- 7 replies
- 12 kudos
create_auto_cdc_from_snapshot_flow Python session resolution fails if having multiple snapshot flows
When a pipeline contains more than one create_auto_cdc_from_snapshot_flow flow (each driven by a custom Python next_snapshot_and_version function), flow resolution fails intermittently/consistently with: RuntimeError: The original Spark session is be...
- 1050 Views
- 7 replies
- 12 kudos
- 12 kudos
Hi again, it seems to have regressed on me, my pipeline is failing again, anybody else seeing this?
- 12 kudos
- 915 Views
- 6 replies
- 10 kudos
Resolved! Photon enabled but a large share of the plan is falling back, cost up and runtime flat
Hi everyone,Trying to work out whether this is expected or whether I have misconfigured something.We enabled Photon on a job cluster running a nightly aggregation over roughly 2TB. The expectation was the usual improvement. What we got instead was ru...
- 915 Views
- 6 replies
- 10 kudos
- 10 kudos
Great questions! Would love to hear from someone who’s dealt with Photon fallbacks, especially around Python UDFs. Curious to know if rewriting them actually made a noticeable difference in runtime and cost.
- 10 kudos
- 21655 Views
- 11 replies
- 2 kudos
Resolved! Performance enhancement while writing dataframes into Parquet tables
Hi,I am trying to write the contents of a dataframe into a parquet table using the command below.df.write.mode("overwrite").format("parquet").saveAsTable("sample_parquet_table")The dataframe contains an extract from one of our source systems, which h...
- 21655 Views
- 11 replies
- 2 kudos
- 2 kudos
Writing an effective essay takes more than putting ideas on paper; it requires solid research, logical structure, clear arguments, and careful proofreading. Students working on challenging assignments may find essay writing help useful for organizing...
- 2 kudos
- 830 Views
- 7 replies
- 4 kudos
Resolved! Compatibility Mode for AWS Redshift
Hello,I have several tables loaded from S3 using Lakeflow pipelines which generates streaming table and materialized views. Want to be able to read these tables in AWS redshift.Based on the Databricks documentation for compatibility mode:Databricks C...
- 830 Views
- 7 replies
- 4 kudos
- 4 kudos
Thanks @ivanvyd , from describe extended table, i can see metadata location as s3-compat, detailed table information has streaming table, location s3-uc. In theory, the databricks unity catalog connector with AWS should read metadata from the s3-comp...
- 4 kudos
- 722 Views
- 4 replies
- 6 kudos
Resolved! At what data size do you stop reaching for Spark?
Hi everyone,A question I keep having with my team and I would like to hear how others think about it.A lot of the jobs we run are not big. Plenty of our pipelines process a few gigabytes, some considerably less. We run them on Spark because that is w...
- 722 Views
- 4 replies
- 6 kudos
- 6 kudos
Is there a size threshold?Most teams that have thought hard about this land somewhere around "if a job's working set fits comfortably in memory on one large-ish node and doesn't need to scale unpredictably, Spark's distributed machinery is pure overh...
- 6 kudos
- 11270 Views
- 12 replies
- 14 kudos
Resolved! Databricks Certified Professional Data Scientist Exam Question Types
Hello,I am not seeing a lot of information regarding the Databricks Certified Professional Data Scientistexam. I took the Associate Developer in Apache Spark Exam last year and the materials for the exam seemed much more focused than what I found for...
- 11270 Views
- 12 replies
- 14 kudos
- 14 kudos
Could you please share the resources need to clear this certification? can you please help
- 14 kudos
- 589 Views
- 2 replies
- 2 kudos
Resolved! Streaming read fails with "Detected a data update" after a restatement job touches the source
Hi everyone,Looking for the right pattern here rather than a workaround.Setup. DBR 15.4 LTS, Unity Catalog. A silver streaming table reads from a bronze Delta table with a normal streaming read. Bronze is append only in the ordinary course of busines...
- 589 Views
- 2 replies
- 2 kudos
- 2 kudos
This is a classic "Streaming vs. Batch" impedance mismatch. You are encountering the "Data change detected" exception because standard Spark Structured Streaming (and by extension, the DLT/Lakeflow logic) assumes the source is an append-only event l...
- 2 kudos
- 20383 Views
- 2 replies
- 2 kudos
ADD COLUMN IF NOT EXISTS does not recognize "IF NOT EXIST". How do I add a column to an existing delta table with SQL if the column does not already exist?
How do I add a column to an existing delta table with SQL if the column does not already exist?I am using the following code: <%sqlALTER TABLE table_name ADD COLUMN IF NOT EXISTS column_name type; >but it prints the error: <[PARSE_SYNTAX_ERROR] Synta...
- 20383 Views
- 2 replies
- 2 kudos
- 2 kudos
Here some SQL devilry that does not require SQL Scripting nor pythons. It adds a my_extra_column FLOAT to the target table with null values. I'm as shocked as anyone that this works, (and if it's undocumented behavior, someone in Databricks Command ...
- 2 kudos
- 205 Views
- 0 replies
- 0 kudos
Login Error Assistance - Reference ID: a728332e-daf7-436d-a676-99ac4b3b0033
Unable to log in to the Databricks account to opt for the subscription to use the compute options, but in Azure, able to create workflows.
- 205 Views
- 0 replies
- 0 kudos
- 514 Views
- 2 replies
- 3 kudos
Resolved! Best practices for SLA monitoring and automated retries across hundreds of Lakeflow Jobs
Hi everyone,We are in the process of migrating our legacy Airflow DAGs (largely time-based cron schedules) to Lakeflow Jobs, and I want to fully leverage the platform's event-driven capabilities rather than just replicating the old "timer-based" patt...
- 514 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi,Migrating the same way. The messy parts.On file arrival triggers, first check whether your external locations have file events enabled, because the behaviour differs sharply. Without them you get a cap of 50 triggered jobs per workspace and at mos...
- 3 kudos
- 775 Views
- 4 replies
- 5 kudos
Resolved! Non-deterministic ROW_NUMBER() results across Unity Catalog environments
We have a notebook that calculates the "mode" (most frequent value) of a customer_parent_name field using ROW_NUMBER() OVER (PARTITION BY ... ORDER BY count DESC). The exact same notebook code is deployed to two Unity Catalog-governed environments (D...
- 775 Views
- 4 replies
- 5 kudos
- 5 kudos
Hi,Your diagnosis is correct, and the fix is simpler than you are expecting.On question one, yes, this is expected and it is not a Databricks or Unity Catalog behaviour. ROW_NUMBER with an ORDER BY that does not define a total order is non determinis...
- 5 kudos
- 517 Views
- 2 replies
- 3 kudos
Resolved! igrating from Cron-based Airflow to Lakeflow's Data-Aware Triggers — Real-world experiences?
Hi everyone,We are in the process of migrating our legacy Airflow DAGs (largely time-based cron schedules) to Lakeflow Jobs, and I want to fully leverage the platform's event-driven capabilities rather than just replicating the old "timer-based" patt...
- 517 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi,Migrating the same way. The messy parts.On file arrival triggers, first check whether your external locations have file events enabled, because the behaviour differs sharply. Without them you get a cap of 50 triggered jobs per workspace and at mos...
- 3 kudos
- 13710 Views
- 12 replies
- 4 kudos
Resolved! ImportError: cannot import name 'AnalyzeArgument' from 'pyspark.sql.udtf'
Hello community,I installed databricks extension on my vscode ide. How to fix this error? I created the environment to run locally my notebooks and selected the available remote cluster to execute my notebook, what else?I Have this error: ImportError...
- 13710 Views
- 12 replies
- 4 kudos
- 4 kudos
@unj1m yes, as Alberto said you don't need to install pyspark, it is included in your cluster configuration.
- 4 kudos
- 368 Views
- 2 replies
- 2 kudos
Choosing between a materialized view and a job-maintained Delta table for an expensive SQL query
I am relatively new to Databricks and do not yet have much experience with materialized views, Delta tables, incremental processing, or the computational costs associated with refreshing persisted results.I currently have a Databricks SQL view that c...
- 368 Views
- 2 replies
- 2 kudos
- 2 kudos
Adding to the above, can do one small validation before deciding between MV and a custom Delta/MERGE solution.Databricks now lets you check whether the MV query is structurally eligible for incremental refresh:EXPLAIN CREATE MATERIALIZED VIEW mv_orde...
- 2 kudos
- 365 Views
- 2 replies
- 0 kudos
Are attached-but-idle serverless notebooks billed? Idle timeout? 32 min job, 86 min billed
Hi all, hope everyone is doing well. Two questions I am hoping someone can help with: is there a way to set an idle timeout for serverless notebooks, and is an attached-but-idle serverless notebook billed while it sits there?Some background on why I ...
- 365 Views
- 2 replies
- 0 kudos
- 0 kudos
@Louis_Frolio Thanks, this clarifies the idle billing behaviour.I also observed a very similar pattern in another Azure workspace using the EU West serverless all-purpose SKU. The last visible notebook execution was around 09:56 UTC, while system.bil...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
