- 3965 Views
- 3 replies
- 0 kudos
Get information from Power BI via XMLA
Hello everyone I am trying to get information from Power BI semantic models via XMLA endpoint using PySpark in Databricks.Can someone help me with that?tks
- 3965 Views
- 3 replies
- 0 kudos
- 0 kudos
You would need the power bi report to be in a Fabric/premium capacity workspace and Capacity setting XMLA Endpoint = Read (or Read/Write) Any specific reason why via XMLA endpoint?Via the REST API : Datasets - Execute Queries - REST API (Power BI Pow...
- 0 kudos
- 36 Views
- 0 replies
- 0 kudos
Oracle CDC Ingestion Pipeline - Schema Exploration finds 0 tables when DB_DOMAIN
Hi,I'm setting up an Oracle Integrated CDC Managed Ingestion Pipeline (LakeFlow Connect) on Azure Databricks. The pipeline connects to Oracle 19c successfully but Schema Exploration always returns 0 tables.Error:Schema Exploration has COMPLETED in 5 ...
- 36 Views
- 0 replies
- 0 kudos
- 206 Views
- 0 replies
- 0 kudos
Are We Entering the Context Engineering Era?
I’ve been thinking about how enterprise AI is changing, especially after attending the Data + AI Summit 2026 and hearing Ali Ghodsi emphasize the importance of enterprise context. It made the direction much clearer to me. We already have very capable...
- 206 Views
- 0 replies
- 0 kudos
- 269 Views
- 3 replies
- 0 kudos
How to extract data from SAP to Databricks?
Hey everyone, I'm working through a project where I need to get data out of SAP S4Hana and into Databricks, and I'd love to pick the brains of anyone who's done this before.A few things I'm trying to figure out:How are you actually pulling the data o...
- 269 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi,SAP BDC is one option to consider.SAP Databricks in Business Data Cloud sap-bdc OpenSharing
- 0 kudos
- 233 Views
- 2 replies
- 0 kudos
Error DELTA_CATALOG_MANAGED_TABLE_UPGRADE_WITH_OTHER_PROPERTIES during catalog commit upgrade
Hi everyone,I'm encountering an issue with the Catalog Commit functionality while attempting to upgrade a table, following the official Databricks documentationCatalog commits | Databricks on AWSWhen running the following command on a Managed Iceberg...
- 233 Views
- 2 replies
- 0 kudos
- 0 kudos
This looks like an interaction between the catalog-managed protocol upgrade and VARIANT shredding, rather than an issue with the SQL syntax itself.delta.feature.catalogManaged needs to be enabled as a standalone table/protocol upgrade. In this case, ...
- 0 kudos
- 5504 Views
- 3 replies
- 1 kudos
Reading Excel files with Spark returns formula values instead of computed values
Hi,I'm seeing inconsistent behavior when reading Excel files using the built-in connector Lakeflow Connector with spark.read.format("excel") (doc: https://docs.databricks.com/aws/en/query/formats/excel). I read an .xlsx file from S3 using this functi...
- 5504 Views
- 3 replies
- 1 kudos
- 1 kudos
This is admittedly partially an advertisement, since I wrote it, but I think it's useful here, so I'll share it.My team and I have implemented (what I believe to be) the first bug-for-bug, bit-perfect math, calculation engine for xlsx files. It works...
- 1 kudos
- 179 Views
- 0 replies
- 1 kudos
why micro-batching matters so much in Databricks Auto Loader and Structured Streaming
I recently spent some time understanding why micro-batching matters so much in Databricks Auto Loader and Structured Streaming, and it changed the way I look at streaming pipelines.Earlier, I used to think streaming meant processing every record the ...
- 179 Views
- 0 replies
- 1 kudos
- 267 Views
- 1 replies
- 1 kudos
Resolved! Lakebase synced table doesn’t recognize Auto CDF on a SDP materialized view
I’m trying to create a triggered Lakebase synced table from a SDP-created materialized view.The source MV uses Automatic CDF: row tracking is enabled and legacy CDF (delta.enableChangeDataFeed) is disabled. The Automatic CDF workspace preview is enab...
- 267 Views
- 1 replies
- 1 kudos
- 1 kudos
Short answer: No — Lakebase synced tables in Triggered/Continuous mode do not currently support Automatic CDF from materialized views. This is a documented limitation, not a configuration issue.The synced tables documentation explicitly states:"Sourc...
- 1 kudos
- 197 Views
- 1 replies
- 0 kudos
Does enabling variantType-preview disable file-level data skipping for columns after a VARIANT colum
I have a Delta table (Unity Catalog, liquid clustering) where file-level data skipping never prunes files — a narrow predicate on the clustered timestamp column still reads every file (Files pruned = 0), even though row-group skipping works. The tabl...
- 197 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Alex13 , thanks for the detailed minimal repro. Isolating variantType-preview as the trigger was the hard part, and it makes your three questions answerable. Here is my read. Is this expected? Not according to anything public. The only documente...
- 0 kudos
- 320 Views
- 4 replies
- 0 kudos
How should schema evolution be handled across silver and gold layers in a medallion architecture?
We run a medallion pipeline on Databricks:Bronze: AutoLoader ingests raw CSV files into Delta tables (append-only, all columns as STRING, schema evolution via addNewColumns)Silver: PySpark jobs clean and transform bronze data into Delta tables using ...
- 320 Views
- 4 replies
- 0 kudos
- 0 kudos
Schema evolution across Silver and Gold layers should ideally be handled with clear contracts between each layer rather than allowing changes to propagate automatically.For the Silver layer, it usually makes sense to support controlled schema changes...
- 0 kudos
- 21609 Views
- 4 replies
- 2 kudos
Resolved! ThreadPoolExecutor in Databricks
I am using a threadpool executor and running notebooks in parallel. However, these parallel notebooks are not using executors at all and all the load is going towards the driver node resulting in running out of memory for the driver node and eventual...
- 21609 Views
- 4 replies
- 2 kudos
- 2 kudos
Good point on the threading vs. process distinction. To add a bit morecontext:Root cause: ThreadPoolExecutor only parallelizes within the driver'ssingle Python process. All threads share the same driver memory, sopandas DataFrame creation across thre...
- 2 kudos
- 3161 Views
- 3 replies
- 2 kudos
Best sequence of using Vacuum, optimize, fsck repair and refresh commands.
I have a delta table whose size will increases gradually now we have around 1.5 crores of rows while running vacuum command on that table i am getting the below error.ERROR: Job aborted due to stage failure: Task 7 in stage 491.0 failed 4 times, most...
- 3161 Views
- 3 replies
- 2 kudos
- 2 kudos
The executor heartbeat timeout makes sense to investigate first, especially with a table that has grown to around 1.5 crore rows. I’d be interested to know whether increasing executor resources actually resolved the issue before changing the command ...
- 2 kudos
- 259 Views
- 2 replies
- 3 kudos
Resolved! Data quality Lineage Root cause analysis
Hi everyone,I'm working on a client use case where they want to visualize data quality together with data lineage.My current approach is to combine:Unity Catalog Data Lineage for upstream/downstream dependenciesLakehouse Monitoring (or other data mon...
- 259 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Niyojit ,I think your proposed architecture is broadly the right direction, but I would separate lineage collection from lineage visualization.Unity Catalog already exposes table-to-table relationships programmatically through system.access.tabl...
- 3 kudos
- 247 Views
- 3 replies
- 0 kudos
Resolved! Skipping malformed records when reading Avro-files
Using read_files to read Avro-formatted blobs from an external location, I get the error:> Malformed records are detected in record parsing: Event 0. Parse Mode: FAILFAST. To process malformed records as null result, try setting the option 'mode' as ...
- 247 Views
- 3 replies
- 0 kudos
- 0 kudos
Thanks for the workaround!That said, I do think this is if nothing else, a documentation issue with Databricks.Documentation says:> For options specific to each file format (JSON, CSV, XML, Parquet, Avro, text, ORC, and binary), see DataFrameReader o...
- 0 kudos
- 252 Views
- 3 replies
- 1 kudos
Unity Catalog service credential get_token rejects api:// scope format — "not a valid URI"
Calling get_token() on a Unity Catalog service credential fails for any scope using the api:// App ID URI format. Only https://-scheme resource scopes succeed. The same api:// scopes work correctly with a service principal.Reproductioncredential = db...
- 252 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Oumeima ,Totally understand. Let's hope support comes through with a fix
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
