- 16 Views
- 1 replies
- 1 kudos
AUTO CDC FROM SNAPSHOT schema evolution in history tracking columns
Does the create_auto_cdc_from_snapshot_flow() allow for expansion of included columns in track_history_column_list or track_history_except_column_list without doing a full refresh? My intended flow is: Bronze - Managed ingestion of table (scd 1) usin...
- 16 Views
- 1 replies
- 1 kudos
- 1 kudos
@carl_hsg Yes, I tested expanding track_history_column_list on an existing SCD2 target and it did not require a full refresh in my test.The part that seems unclear in the docs is what happens when you expand the tracking list or introduce a new colum...
- 1 kudos
- 82 Views
- 3 replies
- 0 kudos
Failed to resolve External API
I am currently testing on a Premium trial, but my workspace is unable to resolve my external API to pull data. May I know if I need to update anything on the settings side to fix this issue?Thanks!
- 82 Views
- 3 replies
- 0 kudos
- 0 kudos
This is usually related to network access rather than the Premium trial itself. I’d check the workspace’s serverless/external network access settings and make sure the API domain is allowed, then verify DNS resolution and any firewall or IP allowlist...
- 0 kudos
- 121 Views
- 2 replies
- 0 kudos
Dark mode setting ignored when embedding BI dashboard inside Databricks App
When using the '@databricks/aibi-client' library to embed a BI dashboard inside a databricks app, the " colorScheme: 'dark' " setting is ignored when the app's 'prefer-color-scheme' is 'light' by default.As a result, I am unable to set my dashboards ...
- 121 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @DB-RKL Yes I have followed the example in the AIBI client docs.I have verified that the colour scheme will render in dark mode just fine when running my app (Nuxt.js framework) in a local server, but when running it inside a Databricks App server...
- 0 kudos
- 26 Views
- 0 replies
- 0 kudos
Are attached-but-idle serverless notebooks billed? Idle timeout? 32 min job, 86 min billed
Hi all, hope everyone is doing well. Two questions I am hoping someone can help with: is there a way to set an idle timeout for serverless notebooks, and is an attached-but-idle serverless notebook billed while it sits there?Some background on why I ...
- 26 Views
- 0 replies
- 0 kudos
- 196 Views
- 4 replies
- 0 kudos
AI/BI Dashboards
Hi AllMy client is looking to create a few dashboards in DataBricks instead of Power BI. We already have a template in Power BI (header for each page, information about out BI Page) in each of our power bi report which has colors, fonts, images etc.H...
- 196 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @km1837, Databricks AI/BI Dashboards have solid theming and organisational capabilities that should cover most of what you are used to in Power BI. AI/BI Dashboards don't have a "template file" concept the way Power BI does with .pbit files, but y...
- 0 kudos
- 8 Views
- 0 replies
- 0 kudos
Slow Running SQL Query
The following Query is running slowly even though it returns only ~ a million rows. It takes around 25 mins. I am trying Produce a pre-aggregated summary table (grouped by 9 dimension attributes) from a large fact table, to be consumed by a Power BI ...
- 8 Views
- 0 replies
- 0 kudos
- 89 Views
- 3 replies
- 1 kudos
How does AUTO CDC resolve same-timestamp tie in SEQUENCE BY?
We are using AUTO CDC (APPLY CHANGES INTO) to process change data capture events from an upstream source (via Autoloader ingesting CDC files landed in cloud storage) into our Silver layer tables. Our source occasionally delivers out-of-order events —...
- 89 Views
- 3 replies
- 1 kudos
- 1 kudos
@Khasim_1 on your third question, check delete-tombstone retention alongside the sequencing change. The default is two days. The create_auto_cdc_flow reference gives this instruction:Set pipelines.cdc.tombstoneGCThresholdInSeconds to a value that exc...
- 1 kudos
- 125 Views
- 3 replies
- 3 kudos
Non-deterministic ROW_NUMBER() results across Unity Catalog environments
We have a notebook that calculates the "mode" (most frequent value) of a customer_parent_name field using ROW_NUMBER() OVER (PARTITION BY ... ORDER BY count DESC). The exact same notebook code is deployed to two Unity Catalog-governed environments (D...
- 125 Views
- 3 replies
- 3 kudos
- 3 kudos
@Khasim_1 another option for calculating the most frequent name is mode(customer_parent_name, true). The second argument requests deterministic tie handling and is supported in Databricks SQL and DBR 14.1+.SELECT duns_number, mode(customer_pa...
- 3 kudos
- 10809 Views
- 11 replies
- 14 kudos
Resolved! Databricks Certified Professional Data Scientist Exam Question Types
Hello,I am not seeing a lot of information regarding the Databricks Certified Professional Data Scientistexam. I took the Associate Developer in Apache Spark Exam last year and the materials for the exam seemed much more focused than what I found for...
- 10809 Views
- 11 replies
- 14 kudos
- 14 kudos
Thanks! I recently passed my exam with 93%. I really struggled to find good preparation resources, but I found authentic practice tests Certs-topic It worked so well for me.
- 14 kudos
- 206 Views
- 4 replies
- 0 kudos
Auto Loader stream fails on RocksDB checkpoint after enabling managed file events
Hello Community.We recently enabled the following option on several existing Auto Loader streams:"cloudFiles.useManagedFileEvents": "true"Most streams continued working, but one now fails while restoring its existing checkpoint, referencing a file si...
- 206 Views
- 4 replies
- 0 kudos
- 0 kudos
After further investigation into it, have understood the root cause and the fix for it.After enabling Auto Loader managed file events across the platform: .option("cloudFiles.useManagedFileEvents", "true")the failed stream started discovering RocksDB...
- 0 kudos
- 170 Views
- 3 replies
- 0 kudos
CSV export from AI/BI Dashboard table widget does not preserve visual column order
SummaryWhen downloading data as CSV from a table widget on an AI/BI Dashboard (kebab menu → Download → Data → Download CSV), the column order in the exported file does not match the column order displayed in the dashboard. All aggregated (measure) co...
- 170 Views
- 3 replies
- 0 kudos
- 0 kudos
I built a copy of your setup on a current workspace and couldn't get it to fail, which is itself a useful clue. I matched it fairly closely: a plain table, several SUM measures with custom display names, a date column via DATE_TRUNC, columns dragged ...
- 0 kudos
- 92 Views
- 2 replies
- 1 kudos
Compatibility Mode for AWS Redshift
Hello,I have several tables loaded from S3 using Lakeflow pipelines which generates streaming table and materialized views. Want to be able to read these tables in AWS redshift.Based on the Databricks documentation for compatibility mode:Databricks C...
- 92 Views
- 2 replies
- 1 kudos
- 1 kudos
I’ve been wondering about this too. The gap between Compatibility Mode and how Redshift Spectrum actually discovers/queries the Delta data isn’t very clear from the docs. Would love to see a concrete Redshift setup example or recommended architecture...
- 1 kudos
- 130 Views
- 1 replies
- 1 kudos
Storage Credential creation fails with "Access Connector ... could not be found"
Subject: Storage Credential creation fails with "Access Connector ... could not be found" despite fully verified Azure configurationHi all,I'm trying to create a storage credential (Azure Managed Identity) for Unity Catalog on a Premium Azure Databri...
- 130 Views
- 1 replies
- 1 kudos
- 1 kudos
@Data_Engineer14, Great troubleshooting so far. The fact that every Azure-side check passes cleanly but the storage credential creation still fails is a strong signal that this is a backend registration issue, not a misconfiguration on your end. Wh...
- 1 kudos
- 114 Views
- 1 replies
- 1 kudos
Best practices for SLA monitoring and automated retries across hundreds of Lakeflow Jobs
Hi everyone,We are in the process of migrating our legacy Airflow DAGs (largely time-based cron schedules) to Lakeflow Jobs, and I want to fully leverage the platform's event-driven capabilities rather than just replicating the old "timer-based" patt...
- 114 Views
- 1 replies
- 1 kudos
- 1 kudos
Just answer the details in another thread. Please check.
- 1 kudos
- 125 Views
- 1 replies
- 1 kudos
igrating from Cron-based Airflow to Lakeflow's Data-Aware Triggers — Real-world experiences?
Hi everyone,We are in the process of migrating our legacy Airflow DAGs (largely time-based cron schedules) to Lakeflow Jobs, and I want to fully leverage the platform's event-driven capabilities rather than just replicating the old "timer-based" patt...
- 125 Views
- 1 replies
- 1 kudos
- 1 kudos
Thanks for the question. This is a very important and interesting question. Please find details from my side.How are you using File Arrival Triggers and Table Update Triggers to avoid running compute against empty or unchanged sources? Have you seen ...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
