- 101 Views
- 1 replies
- 0 kudos
How to extract data from SAP to Databricks?
Hey everyone, I'm working through a project where I need to get data out of SAP S4Hana and into Databricks, and I'd love to pick the brains of anyone who's done this before.A few things I'm trying to figure out:How are you actually pulling the data o...
- 101 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @RaulTovar,Please go the below thread. Hopeful you'll get your questions answered.https://community.databricks.com/t5/data-engineering/data-migration-from-sap-s-4hana-to-databricks/td-p/147536Also, feel free to check/search on the community as som...
- 0 kudos
- 163 Views
- 2 replies
- 0 kudos
Error DELTA_CATALOG_MANAGED_TABLE_UPGRADE_WITH_OTHER_PROPERTIES during catalog commit upgrade
Hi everyone,I'm encountering an issue with the Catalog Commit functionality while attempting to upgrade a table, following the official Databricks documentationCatalog commits | Databricks on AWSWhen running the following command on a Managed Iceberg...
- 163 Views
- 2 replies
- 0 kudos
- 0 kudos
This looks like an interaction between the catalog-managed protocol upgrade and VARIANT shredding, rather than an issue with the SQL syntax itself.delta.feature.catalogManaged needs to be enabled as a standalone table/protocol upgrade. In this case, ...
- 0 kudos
- 5445 Views
- 3 replies
- 1 kudos
Reading Excel files with Spark returns formula values instead of computed values
Hi,I'm seeing inconsistent behavior when reading Excel files using the built-in connector Lakeflow Connector with spark.read.format("excel") (doc: https://docs.databricks.com/aws/en/query/formats/excel). I read an .xlsx file from S3 using this functi...
- 5445 Views
- 3 replies
- 1 kudos
- 1 kudos
This is admittedly partially an advertisement, since I wrote it, but I think it's useful here, so I'll share it.My team and I have implemented (what I believe to be) the first bug-for-bug, bit-perfect math, calculation engine for xlsx files. It works...
- 1 kudos
- 96 Views
- 0 replies
- 1 kudos
why micro-batching matters so much in Databricks Auto Loader and Structured Streaming
I recently spent some time understanding why micro-batching matters so much in Databricks Auto Loader and Structured Streaming, and it changed the way I look at streaming pipelines.Earlier, I used to think streaming meant processing every record the ...
- 96 Views
- 0 replies
- 1 kudos
- 117 Views
- 1 replies
- 1 kudos
Resolved! Lakebase synced table doesn’t recognize Auto CDF on a SDP materialized view
I’m trying to create a triggered Lakebase synced table from a SDP-created materialized view.The source MV uses Automatic CDF: row tracking is enabled and legacy CDF (delta.enableChangeDataFeed) is disabled. The Automatic CDF workspace preview is enab...
- 117 Views
- 1 replies
- 1 kudos
- 1 kudos
Short answer: No — Lakebase synced tables in Triggered/Continuous mode do not currently support Automatic CDF from materialized views. This is a documented limitation, not a configuration issue.The synced tables documentation explicitly states:"Sourc...
- 1 kudos
- 157 Views
- 1 replies
- 0 kudos
Does enabling variantType-preview disable file-level data skipping for columns after a VARIANT colum
I have a Delta table (Unity Catalog, liquid clustering) where file-level data skipping never prunes files — a narrow predicate on the clustered timestamp column still reads every file (Files pruned = 0), even though row-group skipping works. The tabl...
- 157 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Alex13 , thanks for the detailed minimal repro. Isolating variantType-preview as the trigger was the hard part, and it makes your three questions answerable. Here is my read. Is this expected? Not according to anything public. The only documente...
- 0 kudos
- 255 Views
- 4 replies
- 0 kudos
How should schema evolution be handled across silver and gold layers in a medallion architecture?
We run a medallion pipeline on Databricks:Bronze: AutoLoader ingests raw CSV files into Delta tables (append-only, all columns as STRING, schema evolution via addNewColumns)Silver: PySpark jobs clean and transform bronze data into Delta tables using ...
- 255 Views
- 4 replies
- 0 kudos
- 0 kudos
Schema evolution across Silver and Gold layers should ideally be handled with clear contracts between each layer rather than allowing changes to propagate automatically.For the Silver layer, it usually makes sense to support controlled schema changes...
- 0 kudos
- 21557 Views
- 4 replies
- 2 kudos
Resolved! ThreadPoolExecutor in Databricks
I am using a threadpool executor and running notebooks in parallel. However, these parallel notebooks are not using executors at all and all the load is going towards the driver node resulting in running out of memory for the driver node and eventual...
- 21557 Views
- 4 replies
- 2 kudos
- 2 kudos
Good point on the threading vs. process distinction. To add a bit morecontext:Root cause: ThreadPoolExecutor only parallelizes within the driver'ssingle Python process. All threads share the same driver memory, sopandas DataFrame creation across thre...
- 2 kudos
- 3136 Views
- 3 replies
- 2 kudos
Best sequence of using Vacuum, optimize, fsck repair and refresh commands.
I have a delta table whose size will increases gradually now we have around 1.5 crores of rows while running vacuum command on that table i am getting the below error.ERROR: Job aborted due to stage failure: Task 7 in stage 491.0 failed 4 times, most...
- 3136 Views
- 3 replies
- 2 kudos
- 2 kudos
The executor heartbeat timeout makes sense to investigate first, especially with a table that has grown to around 1.5 crore rows. I’d be interested to know whether increasing executor resources actually resolved the issue before changing the command ...
- 2 kudos
- 125 Views
- 2 replies
- 3 kudos
Resolved! Data quality Lineage Root cause analysis
Hi everyone,I'm working on a client use case where they want to visualize data quality together with data lineage.My current approach is to combine:Unity Catalog Data Lineage for upstream/downstream dependenciesLakehouse Monitoring (or other data mon...
- 125 Views
- 2 replies
- 3 kudos
- 3 kudos
Hi @Niyojit ,I think your proposed architecture is broadly the right direction, but I would separate lineage collection from lineage visualization.Unity Catalog already exposes table-to-table relationships programmatically through system.access.tabl...
- 3 kudos
- 168 Views
- 3 replies
- 0 kudos
Resolved! Skipping malformed records when reading Avro-files
Using read_files to read Avro-formatted blobs from an external location, I get the error:> Malformed records are detected in record parsing: Event 0. Parse Mode: FAILFAST. To process malformed records as null result, try setting the option 'mode' as ...
- 168 Views
- 3 replies
- 0 kudos
- 0 kudos
Thanks for the workaround!That said, I do think this is if nothing else, a documentation issue with Databricks.Documentation says:> For options specific to each file format (JSON, CSV, XML, Parquet, Avro, text, ORC, and binary), see DataFrameReader o...
- 0 kudos
- 186 Views
- 3 replies
- 1 kudos
Unity Catalog service credential get_token rejects api:// scope format — "not a valid URI"
Calling get_token() on a Unity Catalog service credential fails for any scope using the api:// App ID URI format. Only https://-scheme resource scopes succeed. The same api:// scopes work correctly with a service principal.Reproductioncredential = db...
- 186 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @Oumeima ,Totally understand. Let's hope support comes through with a fix
- 1 kudos
- 446 Views
- 6 replies
- 17 kudos
Why is the default auto-termination for serverless interactive notebook compute 60 minutes?
We've been digging into serverless interactive compute costs in notebooks, because our internal cost monitoring (cross-referencing system.billing.usage with system.query.history) showed a large share of the spend coming from idle time rather than act...
- 446 Views
- 6 replies
- 17 kudos
- 17 kudos
Hi @elizeu_reis , great detective work here. Cross-referencing system.billing.usage with system.query.history is exactly the right way to separate real usage from idle burn. I did some digging, and I can confirm most of what you observed. There are t...
- 17 kudos
- 85 Views
- 1 replies
- 0 kudos
Unexpected behavior of Delta VACUUM – need explanation
HelloI have a question about how VACUUM works.We executed the following VACUUM command from a PySpark job:spark.sql("""VACUUM catalog_name.schema_name.table_name RETAIN 168 HOURS""")The Delta table has a table property retention period of 365 days, b...
- 85 Views
- 1 replies
- 0 kudos
- 0 kudos
Greetings @bricks_2026 , I did some digging and here is what I found. First, good question, and you've already pulled the right evidence from the table history. This looks like retention semantics rather than a PySpark quoting issue or a logging bu...
- 0 kudos
- 164 Views
- 3 replies
- 0 kudos
Disabling Change Tracking and enabling Change Data Capture in SQL Server Lakeflow
Hi All,I want to know my lakeflow connect pipeline has stopped running because Change Tracking is enabled and table doesn't have a primary key and i want to switch to CDC now. Will it impact my pipeline. What are the steps i need to follow to on inge...
- 164 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi Upendra,When both Change Tracking and Change Data Capture are enabled the connector generally prioritizes Change Tracking. Since CT requires the source table to have a primary key and table did not have one, it causes the pipeline to fail. Even af...
- 0 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
