- 139 Views
- 3 replies
- 0 kudos
CSV export from AI/BI Dashboard table widget does not preserve visual column order
SummaryWhen downloading data as CSV from a table widget on an AI/BI Dashboard (kebab menu → Download → Data → Download CSV), the column order in the exported file does not match the column order displayed in the dashboard. All aggregated (measure) co...
- 139 Views
- 3 replies
- 0 kudos
- 0 kudos
I built a copy of your setup on a current workspace and couldn't get it to fail, which is itself a useful clue. I matched it fairly closely: a plain table, several SUM measures with custom display names, a date column via DATE_TRUNC, columns dragged ...
- 0 kudos
- 79 Views
- 1 replies
- 0 kudos
Dark mode setting ignored when embedding BI dashboard inside Databricks App
When using the '@databricks/aibi-client' library to embed a BI dashboard inside a databricks app, the " colorScheme: 'dark' " setting is ignored when the app's 'prefer-color-scheme' is 'light' by default.As a result, I am unable to set my dashboards ...
- 79 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey @chie_789 Have you tried setting the colour scheme using javascript? we have an example in the AIBI client docs?https://www.jsdelivr.com/package/npm/@databricks/aibi-client
- 0 kudos
- 60 Views
- 2 replies
- 1 kudos
Compatibility Mode for AWS Redshift
Hello,I have several tables loaded from S3 using Lakeflow pipelines which generates streaming table and materialized views. Want to be able to read these tables in AWS redshift.Based on the Databricks documentation for compatibility mode:Databricks C...
- 60 Views
- 2 replies
- 1 kudos
- 1 kudos
I’ve been wondering about this too. The gap between Compatibility Mode and how Redshift Spectrum actually discovers/queries the Delta data isn’t very clear from the docs. Would love to see a concrete Redshift setup example or recommended architecture...
- 1 kudos
- 80 Views
- 2 replies
- 0 kudos
Non-deterministic ROW_NUMBER() results across Unity Catalog environments
We have a notebook that calculates the "mode" (most frequent value) of a customer_parent_name field using ROW_NUMBER() OVER (PARTITION BY ... ORDER BY count DESC). The exact same notebook code is deployed to two Unity Catalog-governed environments (D...
- 80 Views
- 2 replies
- 0 kudos
- 0 kudos
Agree with the explanation from @ShamenParis. Databricks documents this explicitly too for ROW_NUMBER(): If the ordering is not unique, the result is non-deterministic.So if two customer_parent_name values have the same cnt, either one can receive ro...
- 0 kudos
- 17 Views
- 0 replies
- 0 kudos
How does AUTO CDC resolve same-timestamp tie in SEQUENCE BY?
We are using AUTO CDC (APPLY CHANGES INTO) to process change data capture events from an upstream source (via Autoloader ingesting CDC files landed in cloud storage) into our Silver layer tables. Our source occasionally delivers out-of-order events —...
- 17 Views
- 0 replies
- 0 kudos
- 111 Views
- 1 replies
- 1 kudos
Storage Credential creation fails with "Access Connector ... could not be found"
Subject: Storage Credential creation fails with "Access Connector ... could not be found" despite fully verified Azure configurationHi all,I'm trying to create a storage credential (Azure Managed Identity) for Unity Catalog on a Premium Azure Databri...
- 111 Views
- 1 replies
- 1 kudos
- 1 kudos
@Data_Engineer14, Great troubleshooting so far. The fact that every Azure-side check passes cleanly but the storage credential creation still fails is a strong signal that this is a backend registration issue, not a misconfiguration on your end. Wh...
- 1 kudos
- 99 Views
- 1 replies
- 0 kudos
Best practices for SLA monitoring and automated retries across hundreds of Lakeflow Jobs
Hi everyone,We are in the process of migrating our legacy Airflow DAGs (largely time-based cron schedules) to Lakeflow Jobs, and I want to fully leverage the platform's event-driven capabilities rather than just replicating the old "timer-based" patt...
- 99 Views
- 1 replies
- 0 kudos
- 0 kudos
Just answer the details in another thread. Please check.
- 0 kudos
- 107 Views
- 1 replies
- 0 kudos
igrating from Cron-based Airflow to Lakeflow's Data-Aware Triggers — Real-world experiences?
Hi everyone,We are in the process of migrating our legacy Airflow DAGs (largely time-based cron schedules) to Lakeflow Jobs, and I want to fully leverage the platform's event-driven capabilities rather than just replicating the old "timer-based" patt...
- 107 Views
- 1 replies
- 0 kudos
- 0 kudos
Thanks for the question. This is a very important and interesting question. Please find details from my side.How are you using File Arrival Triggers and Table Update Triggers to avoid running compute against empty or unchanged sources? Have you seen ...
- 0 kudos
- 286 Views
- 3 replies
- 2 kudos
Resolved! Databricks Apps - Deployment process - Need help !!
Hi all,Do we have any guidelines or a CI/CD approach for deploying a Databricks app from one environment (e.g., DEV) to PROD?We are currently facing the following challenge:We created a Databricks app in the DEV environment through the UI. At runtime...
- 286 Views
- 3 replies
- 2 kudos
- 2 kudos
Thank you for the details. Yes, I agree we should adopt the DABs for the deployment and define the resources while creating the app ( which can be consider as as solution). However if the permission module is managed separately, lets say via terrafor...
- 2 kudos
- 243 Views
- 2 replies
- 3 kudos
Resolved! Best Practice for Handling Schema Evolution with Auto Loader in Production?
Hi Databricks Community,I’d like to hear how other Data Engineers are handling schema evolution with Auto Loader in production environments.Consider the following scenario:We have a continuously running ingestion pipeline using Auto Loader that proce...
- 243 Views
- 2 replies
- 3 kudos
- 3 kudos
You can absolutely leverage Databricks Genie and the new Genie Ontology to handle downstream changes caused by addNewColumnsWithTypeWidening, and handle the changes using Genie ontology, but it requires understanding the division of labor between you...
- 3 kudos
- 173 Views
- 3 replies
- 0 kudos
Auto Loader stream fails on RocksDB checkpoint after enabling managed file events
Hello Community.We recently enabled the following option on several existing Auto Loader streams:"cloudFiles.useManagedFileEvents": "true"Most streams continued working, but one now fails while restoring its existing checkpoint, referencing a file si...
- 173 Views
- 3 replies
- 0 kudos
- 0 kudos
I am really glad to hear you got the stream unblocked.Thank you for sharing the exact error code (FAILED_READ_FILE.NO_HINT). That makes perfect sense—if the RocksDB .zip archive itself is physically unreadable or corrupted at the storage layer, rathe...
- 0 kudos
- 130 Views
- 1 replies
- 0 kudos
OAuth M2M (client-credentials) - getting error with github run
I'm setting up OAuth M2M (client-credentials) authentication for a service principal, as the long-term replacement for PAT-based auth in a GitHub Actions CI/CD workflow. The token exchange itself succeeds and returns a valid, well-formed access token...
- 130 Views
- 1 replies
- 0 kudos
- 0 kudos
further refined this problem with below if anyone has any idea how I could address? thanks. Later in troubleshooting, this same should_change_password error started appearing on a personal-account PAT as well (not just the SP's OAuth token) — so what...
- 0 kudos
- 112 Views
- 2 replies
- 0 kudos
How to create monotonic function to incrementally add obj_id for datasource in pyspark
I have used monotonically_increasing_id() to add unique to datasource .However , this does not add unique id in increasing number ..Its kind of random.How to perform this in pyspark.
- 112 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey So monotonically_increasing_id() only promises the numbers go up and are unique, not that they're neat like 1, 2, 3. Under the hood it bakes the partition number into the ID, so each partition jumps ahead by billions.If you let me know what you'...
- 0 kudos
- 205 Views
- 1 replies
- 2 kudos
Resolved! Designing an Effective "Quarantine" Pattern for Failed DLT Expectations
Hi everyone,I’m building out a production Medallion Architecture using Delta Live Tables (DLT), and I’m refining our strategy for handling data that fails Expectations.While expect_or_drop and expect_or_fail are straightforward, I’m struggling to fi...
- 205 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi,Good questions. A few things that have worked.On the quarantine pattern, the version most people land on first is two flows from the same source, one with the expectations and one with the inverted condition. It works, but it reads the source twic...
- 2 kudos
- 195 Views
- 2 replies
- 4 kudos
Resolved! Managing Service Principal Permissions at Scale: External Locations vs. Managed Volumes
Hi community,I’m currently architecting a multi-workspace environment under Unity Catalog, and I’m looking for the most sustainable way to manage Service Principal access to raw data landing zones.We are debating between using External Locations (sco...
- 195 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi,On question one, I would reframe the choice rather than answer it directly. Managed volumes sit in Databricks controlled storage, which means an external producer cannot write into them through Unity Catalog. If your landing zone is fed by systems...
- 4 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
