- 106 Views
- 3 replies
- 0 kudos
Auto Loader stream fails on RocksDB checkpoint after enabling managed file events
Hello Community.We recently enabled the following option on several existing Auto Loader streams:"cloudFiles.useManagedFileEvents": "true"Most streams continued working, but one now fails while restoring its existing checkpoint, referencing a file si...
- 106 Views
- 3 replies
- 0 kudos
- 0 kudos
I am really glad to hear you got the stream unblocked.Thank you for sharing the exact error code (FAILED_READ_FILE.NO_HINT). That makes perfect sense—if the RocksDB .zip archive itself is physically unreadable or corrupted at the storage layer, rathe...
- 0 kudos
- 73 Views
- 1 replies
- 0 kudos
CSV export from AI/BI Dashboard table widget does not preserve visual column order
SummaryWhen downloading data as CSV from a table widget on an AI/BI Dashboard (kebab menu → Download → Data → Download CSV), the column order in the exported file does not match the column order displayed in the dashboard. All aggregated (measure) co...
- 73 Views
- 1 replies
- 0 kudos
- 0 kudos
I had a go at reproducing this on a current AI/BI build and couldn't get a plain table widget to misbehave. I built a table grouping dimensions plus sum(...) measures, arranged dimensions-first, and the exported CSV kept that order every time. I test...
- 0 kudos
- 91 Views
- 1 replies
- 0 kudos
OAuth M2M (client-credentials) - getting error with github run
I'm setting up OAuth M2M (client-credentials) authentication for a service principal, as the long-term replacement for PAT-based auth in a GitHub Actions CI/CD workflow. The token exchange itself succeeds and returns a valid, well-formed access token...
- 91 Views
- 1 replies
- 0 kudos
- 0 kudos
further refined this problem with below if anyone has any idea how I could address? thanks. Later in troubleshooting, this same should_change_password error started appearing on a personal-account PAT as well (not just the SP's OAuth token) — so what...
- 0 kudos
- 115 Views
- 1 replies
- 2 kudos
Resolved! Best Practice for Handling Schema Evolution with Auto Loader in Production?
Hi Databricks Community,I’d like to hear how other Data Engineers are handling schema evolution with Auto Loader in production environments.Consider the following scenario:We have a continuously running ingestion pipeline using Auto Loader that proce...
- 115 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi @Islam_hoti,This is a central design decision for any production-grade Lakehouse. In my experience architecting petabyte-scale environments, the goal is to minimize "Pipeline Friction" without sacrificing "Data Governance."Here is the "Architectur...
- 2 kudos
- 71 Views
- 1 replies
- 0 kudos
How to create monotonic function to incrementally add obj_id for datasource in pyspark
I have used monotonically_increasing_id() to add unique to datasource .However , this does not add unique id in increasing number ..Its kind of random.How to perform this in pyspark.
- 71 Views
- 1 replies
- 0 kudos
- 0 kudos
Hey So monotonically_increasing_id() only promises the numbers go up and are unique, not that they're neat like 1, 2, 3. Under the hood it bakes the partition number into the ID, so each partition jumps ahead by billions.If you let me know what you'...
- 0 kudos
- 42 Views
- 0 replies
- 0 kudos
Storage Credential creation fails with "Access Connector ... could not be found"
Subject: Storage Credential creation fails with "Access Connector ... could not be found" despite fully verified Azure configurationHi all,I'm trying to create a storage credential (Azure Managed Identity) for Unity Catalog on a Premium Azure Databri...
- 42 Views
- 0 replies
- 0 kudos
- 130 Views
- 1 replies
- 2 kudos
Resolved! Designing an Effective "Quarantine" Pattern for Failed DLT Expectations
Hi everyone,I’m building out a production Medallion Architecture using Delta Live Tables (DLT), and I’m refining our strategy for handling data that fails Expectations.While expect_or_drop and expect_or_fail are straightforward, I’m struggling to fi...
- 130 Views
- 1 replies
- 2 kudos
- 2 kudos
Hi,Good questions. A few things that have worked.On the quarantine pattern, the version most people land on first is two flows from the same source, one with the expectations and one with the inverted condition. It works, but it reads the source twic...
- 2 kudos
- 113 Views
- 2 replies
- 4 kudos
Resolved! Managing Service Principal Permissions at Scale: External Locations vs. Managed Volumes
Hi community,I’m currently architecting a multi-workspace environment under Unity Catalog, and I’m looking for the most sustainable way to manage Service Principal access to raw data landing zones.We are debating between using External Locations (sco...
- 113 Views
- 2 replies
- 4 kudos
- 4 kudos
Hi,On question one, I would reframe the choice rather than answer it directly. Managed volumes sit in Databricks controlled storage, which means an external producer cannot write into them through Unity Catalog. If your landing zone is fed by systems...
- 4 kudos
- 117 Views
- 2 replies
- 3 kudos
Resolved! Migrating Large-Scale Z-Ordered Tables to Liquid Clustering: Strategies for Production Pipelines
Hi everyone,As we look to modernize our Delta tables, I’m evaluating the move from Z-Ordering to Liquid Clustering, particularly for our historical datasets that exceed 500TB.While the benefits of Liquid Clustering (avoiding over-partitioning and bet...
- 117 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi,The good news is that the migration is much cheaper than you are assuming, because the full rewrite is optional.When you run ALTER TABLE with CLUSTER BY, existing data is not rewritten. Subsequent writes and normal OPTIMIZE runs use the new cluste...
- 3 kudos
- 114 Views
- 1 replies
- 0 kudos
Metric views materialization failure
Hi, I have metric views with materilization option, it was running fine but today it just failed without any change to my metric view definition, all metric views are failing so I guess is something general. Based on limited logs (users can´t fully a...
- 114 Views
- 1 replies
- 0 kudos
- 0 kudos
This error happens because Delta's Liquid Clustering requires column statistics to be collected for every column specified in your cluster_by.cols list (Travel Month, Operating Carrier, Departure Airport IATA, Arrival Airport IATA), but Enzyme (the i...
- 0 kudos
- 105 Views
- 2 replies
- 0 kudos
Oracle NUMBER → DecimalType(38,10) on ingestion, and silver layer best practices
Hello everyone,I've just started a new position and my goal is to build a Data Lakehouse with Databricks. I have no previous experience with Databricks and I'm being helped by an external company. The goal is to centralize all our data in Databricks....
- 105 Views
- 2 replies
- 0 kudos
- 0 kudos
@faruk Handling Oracle NUMBER in the ingestion pipelineThe Oracle JDBC driver maps an NUMBER (no precision, no scale) to DecimalType(38,10) as it must pick a fixed scale for a type system that has no equivalent to Oracle's unconstrained decimal. You ...
- 0 kudos
- 199 Views
- 8 replies
- 2 kudos
shutil.copy from /local_disk0 to Unity Catalog Volume hangs for hours — recommended pattern for log
SetupSingleton Python logger (logging.FileHandler) writing .log files during ADF-orchestrated notebook runs (one notebook per file). Files need to land in a UC Volume.Context on the migration path — this workflow has been reshaped repeatedly by the m...
- 199 Views
- 8 replies
- 2 kudos
- 2 kudos
Hi @DoTA , I have been using a singleton logger as there are helper classes being used within the notebook which will also need to use the logger.
- 2 kudos
- 20811 Views
- 23 replies
- 4 kudos
spotify API get token - raw_input was called, but this frontend does not support input requests.
hello everyone, I'm trying use spotify's api to analyse my music data, but i'm receiving a error during authentication, specifically when I try get the token, above my code.Is it a databricks bug?pip install spotipyfrom spotipy.oauth2 import SpotifyO...
- 20811 Views
- 23 replies
- 4 kudos
- 4 kudos
Roblox scripting can be easier to explore when useful information about tools, features, and compatibility is available in one place. Delta Executor provides resources covering Lua scripts, setup guides, platform support, and updates for users intere...
- 4 kudos
- 186 Views
- 5 replies
- 0 kudos
Agent outside databricks communication with databricks delta table
Hello community,I have following use case in my project:User[ Ask any query in simple english related to data] -> AI Agent -> Databricks unity catalog -> Delta table.Currently required data for project is in volume of workspace. Then we apply medalli...
- 186 Views
- 5 replies
- 0 kudos
- 0 kudos
@Kushal_2612 The slowness you are seeing is generally when an external service submits queries against standard workspace compute or interactive clusters, which carry heavy execution overhead. For querying Unity Catalog Delta tables from an external ...
- 0 kudos
- 114 Views
- 3 replies
- 0 kudos
Handling New Columns in a Databricks Data Pipeline
Hi everyone,I have a question about handling schema changes during data ingestion in Databricks.Let’s say an e-commerce company receives customer data every day with the following columns:Customer_id, name, cityAfter a few months, the source system a...
- 114 Views
- 3 replies
- 0 kudos
- 0 kudos
@gowri_databrick Auto Loader automatically detects and handles new columns when ingesting files. When your source adds the email column, Auto Loader's checkpoint tracks the schema change and can either rescue unknown columns into _rescued_data (safe ...
- 0 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
