- 219 Views
- 3 replies
- 0 kudos
How to extract data from SAP to Databricks?
Hey everyone, I'm working through a project where I need to get data out of SAP S4Hana and into Databricks, and I'd love to pick the brains of anyone who's done this before.A few things I'm trying to figure out:How are you actually pulling the data o...
- 219 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi,SAP BDC is one option to consider.SAP Databricks in Business Data Cloud sap-bdc OpenSharing
- 0 kudos
- 5489 Views
- 3 replies
- 1 kudos
Reading Excel files with Spark returns formula values instead of computed values
Hi,I'm seeing inconsistent behavior when reading Excel files using the built-in connector Lakeflow Connector with spark.read.format("excel") (doc: https://docs.databricks.com/aws/en/query/formats/excel). I read an .xlsx file from S3 using this functi...
- 5489 Views
- 3 replies
- 1 kudos
- 1 kudos
This is admittedly partially an advertisement, since I wrote it, but I think it's useful here, so I'll share it.My team and I have implemented (what I believe to be) the first bug-for-bug, bit-perfect math, calculation engine for xlsx files. It works...
- 1 kudos
- 151 Views
- 0 replies
- 1 kudos
why micro-batching matters so much in Databricks Auto Loader and Structured Streaming
I recently spent some time understanding why micro-batching matters so much in Databricks Auto Loader and Structured Streaming, and it changed the way I look at streaming pipelines.Earlier, I used to think streaming meant processing every record the ...
- 151 Views
- 0 replies
- 1 kudos
- 182 Views
- 1 replies
- 0 kudos
Does enabling variantType-preview disable file-level data skipping for columns after a VARIANT colum
I have a Delta table (Unity Catalog, liquid clustering) where file-level data skipping never prunes files — a narrow predicate on the clustered timestamp column still reads every file (Files pruned = 0), even though row-group skipping works. The tabl...
- 182 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Alex13 , thanks for the detailed minimal repro. Isolating variantType-preview as the trigger was the hard part, and it makes your three questions answerable. Here is my read. Is this expected? Not according to anything public. The only documente...
- 0 kudos
- 290 Views
- 4 replies
- 0 kudos
How should schema evolution be handled across silver and gold layers in a medallion architecture?
We run a medallion pipeline on Databricks:Bronze: AutoLoader ingests raw CSV files into Delta tables (append-only, all columns as STRING, schema evolution via addNewColumns)Silver: PySpark jobs clean and transform bronze data into Delta tables using ...
- 290 Views
- 4 replies
- 0 kudos
- 0 kudos
Schema evolution across Silver and Gold layers should ideally be handled with clear contracts between each layer rather than allowing changes to propagate automatically.For the Silver layer, it usually makes sense to support controlled schema changes...
- 0 kudos
- 201 Views
- 3 replies
- 0 kudos
Resolved! Skipping malformed records when reading Avro-files
Using read_files to read Avro-formatted blobs from an external location, I get the error:> Malformed records are detected in record parsing: Event 0. Parse Mode: FAILFAST. To process malformed records as null result, try setting the option 'mode' as ...
- 201 Views
- 3 replies
- 0 kudos
- 0 kudos
Thanks for the workaround!That said, I do think this is if nothing else, a documentation issue with Databricks.Documentation says:> For options specific to each file format (JSON, CSV, XML, Parquet, Avro, text, ORC, and binary), see DataFrameReader o...
- 0 kudos
- 70 Views
- 0 replies
- 0 kudos
Declarative Until It Isn't: Four Sharp Edges of Lakeflow Declarative Pipelines
The common case is almost too easy — which is exactly why the real engineering lives at the boundaries where the abstraction leaks.Declarative pipelines make the common case feel almost suspiciously easy. You describe the tables you want, point them ...
- 70 Views
- 0 replies
- 0 kudos
- 197 Views
- 1 replies
- 0 kudos
Unable to Enable Unity Catalog – Azure Managed Identity Credential Not Found
Hi Databricks Community,I am trying to enable Unity Catalog in my Azure Databricks account. As part of the setup, I created an Azure Databricks Access Connector and assigned its managed identity the required permissions on my ADLS Gen2 storage accoun...
- 197 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Shubh5073 ,As per my understanding of the issue you are facing, the error indicates that Databricks cannot locate the Access Connector resource in Azure, rather than a problem with your ADLS permissions (since you already correctly assigned Stora...
- 0 kudos
- 250 Views
- 1 replies
- 2 kudos
Is it safe to expose JWT in Databricks Job?
There is an usecase where I want to verify/extract the job caller identity in the job runtime of databricks. The job run_as and creator parameter seems to be fixed at the job configuration and doesn't reflect/update who actually triggered a job run.A...
- 250 Views
- 1 replies
- 2 kudos
- 2 kudos
Hello !What I can say is that your conclusion is correct since DBKS does not currently expose the identity of the principal that called run-now to the running job through the Jobs API or notebook context and the audit event is the authoritative sourc...
- 2 kudos
- 193 Views
- 1 replies
- 0 kudos
Power BI Refresh Fails with "Unable to Continue Fetch After Reconnect" from Azure Databricks
Hi everyone,I'm refreshing a report with dataset (~10 million rows) from Azure Databricks in power Bi service. The refresh runs for about an hour and then fails with: Unable to continue fetch after reconnect. Retry limit exceeded. I am currently unab...
- 193 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Raj,Query execution on Databricks completed in under a second but the failure is occurring during the client-side result streaming phase between Power BI and Azure Databricks.When importing Power BI streams result chunks over HTTP connections. If ...
- 0 kudos
- 314 Views
- 2 replies
- 2 kudos
Serverless Scala JAR: foreachBatch fails with RST_STREAM PROTOCOL_ERROR
Hi everyone,We're migrating a Scala Structured Streaming application (Kinesis → Delta) from classic compute to Databricks Serverless Jobs for cost optimization. During the migration, we're consistently seeing what appears to be a Spark Connect / gRPC...
- 314 Views
- 2 replies
- 2 kudos
- 2 kudos
On (1): the docs don't gate Scala foreachBatch on serverless. Limitations with Databricks Connect for Scala lists streaming foreachBatch as unavailable only on Databricks Runtime 13.3 LTS and below, and serverless limitations names only Trigger.Proce...
- 2 kudos
- 4618 Views
- 10 replies
- 3 kudos
Resolved! Unable to create secret scope -"Fetch request failed due expired user session"
Hi everyone,I’m trying to create an Azure Key Vault-backed secret scope in a Databricks Premium workspace, but I keep getting this error: Fetch request failed due expired user sessionSetup details:Databricks workspace: PremiumAzure Key Vault: Owner p...
- 4618 Views
- 10 replies
- 3 kudos
- 3 kudos
Hi @lofidev I resolved by requesting a policy exemption window to create scopes. The issue was due to a restrictive policy. Precisely what @lingareddy_Alva summarized earlier on this thread "When Databricks tries to programmatically grant its SP Get/...
- 3 kudos
- 526 Views
- 4 replies
- 1 kudos
Disable spark transformations inside try except warning
I'm wondering if there's a way to suppress certain errors? I have a lot of spammy yellow underlining from this one warning, and it doesn't seem that the warning behaves properly. Transformation inside try/except block is lazy, fine, I have actions on...
- 526 Views
- 4 replies
- 1 kudos
- 1 kudos
If your action (count(), collect(), show(), etc.) is already inside the try block, this sounds like a false positive. The warning is likely from a Databricks inspection rather than the Flint time-series library. Check if inspection SCPAP005 can be di...
- 1 kudos
- 763 Views
- 6 replies
- 0 kudos
Resolved! VOID column inside STRUCT fails to cast to VARIANT
Let's say I have the following dataframe%sqlCREATE OR REPLACE TEMPORARY VIEW v_temp ASVALUES (NULL, DATE'2025-12-31'), (NULL, DATE'2026-04-12'), (NULL, DATE'2026-06-23') AS T(PAYMENT_DATE, TRANSACTION_DATE);SELECT PAYMENT_DATE, TYPEOF(PAYMENT_DATE), ...
- 763 Views
- 6 replies
- 0 kudos
- 0 kudos
Your screenshot already uses TO_VARIANT_OBJECT; the VOID field inside the STRUCT is what causes the failure. TO_VARIANT_OBJECT requires every field to be convertible to VARIANT, but VOID is not accepted in that nested path. Scalar CAST(PAYMENT_DATE A...
- 0 kudos
- 722 Views
- 3 replies
- 0 kudos
Advanced Learning Festival (15 June - 06 July 2026) - Voucher Eligibility Confirmation
Hello @Jim ,I completed all four modules of Learning Pathway 1 (Associate Data Engineering) for the Advanced Learning Festival on July 6, 2026, the published end date of the event. My Academy account email is [datasci.patel.rupesh@gmail.com]. Screens...
- 722 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi @Ashwin_DSA ,Thank you for the confirmation. Following up as it's now July 17 and I haven't yet received the voucher at the email address associated with my Academy account (I've checked spam/promotions folders as well).For reference, I completed ...
- 0 kudos