- 138 Views
- 1 replies
- 1 kudos
cross-region DR in Azure Databricks (24h RPO/RTO)
Hi,We're designing a DR strategy for an Azure Databricks platform and would appreciate guidance on current best practices for achieving approximately 24-hour RPO and RTO across Azure regions.Our platform includes Unity Catalog, DAB, Jobs, Notebooks, ...
- 138 Views
- 1 replies
- 1 kudos
- 1 kudos
Your overall design is sensible, but Databricks’ current recommendation is to use Managed Disaster Recovery when your account is eligible. If Managed DR is unavailable, use an active-passive warm standby with Terraform/DABs, cross-region data replica...
- 1 kudos
- 223 Views
- 2 replies
- 2 kudos
Resolved! databricks SQL UDF in select statement
In the Unity Catalog we can now create/register SQL UDFs. There are two types - one that returns table and other that returns just a value. If the function that returns value is based on the SQL query and joins it would in standard relational databas...
- 223 Views
- 2 replies
- 2 kudos
- 2 kudos
The earlier reply has the mechanism broadly right, and your two questions have clean answers: yes to the first, no to the second. A SQL scalar function whose body is a query is planned as a scalar subquery inside the calling statement. Databricks lab...
- 2 kudos
- 368 Views
- 3 replies
- 0 kudos
Unexpected behavior of Delta VACUUM – need explanation
HelloI have a question about how VACUUM works.We executed the following VACUUM command from a PySpark job:spark.sql("""VACUUM catalog_name.schema_name.table_name RETAIN 168 HOURS""")The Delta table has a table property retention period of 365 days, b...
- 368 Views
- 3 replies
- 0 kudos
- 0 kudos
Greetings @bricks_2026 , glad the first answer helped. Your sliding window observation is sharp, but one correction to the mental model: delta.deletedFileRetentionDuration only drives physical deletion at the moment a VACUUM executes. Between your w...
- 0 kudos
- 405 Views
- 3 replies
- 0 kudos
How to extract data from SAP to Databricks?
Hey everyone, I'm working through a project where I need to get data out of SAP S4Hana and into Databricks, and I'd love to pick the brains of anyone who's done this before.A few things I'm trying to figure out:How are you actually pulling the data o...
- 405 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi,SAP BDC is one option to consider.SAP Databricks in Business Data Cloud sap-bdc OpenSharing
- 0 kudos
- 343 Views
- 2 replies
- 0 kudos
Error DELTA_CATALOG_MANAGED_TABLE_UPGRADE_WITH_OTHER_PROPERTIES during catalog commit upgrade
Hi everyone,I'm encountering an issue with the Catalog Commit functionality while attempting to upgrade a table, following the official Databricks documentationCatalog commits | Databricks on AWSWhen running the following command on a Managed Iceberg...
- 343 Views
- 2 replies
- 0 kudos
- 0 kudos
This looks like an interaction between the catalog-managed protocol upgrade and VARIANT shredding, rather than an issue with the SQL syntax itself.delta.feature.catalogManaged needs to be enabled as a standalone table/protocol upgrade. In this case, ...
- 0 kudos
- 267 Views
- 0 replies
- 1 kudos
why micro-batching matters so much in Databricks Auto Loader and Structured Streaming
I recently spent some time understanding why micro-batching matters so much in Databricks Auto Loader and Structured Streaming, and it changed the way I look at streaming pipelines.Earlier, I used to think streaming meant processing every record the ...
- 267 Views
- 0 replies
- 1 kudos
- 435 Views
- 3 replies
- 0 kudos
How should schema evolution be handled across silver and gold layers in a medallion architecture?
We run a medallion pipeline on Databricks:Bronze: AutoLoader ingests raw CSV files into Delta tables (append-only, all columns as STRING, schema evolution via addNewColumns)Silver: PySpark jobs clean and transform bronze data into Delta tables using ...
- 435 Views
- 3 replies
- 0 kudos
- 0 kudos
You can skip using traditional schema migration tools like Liquibase, Flyway or Alembic into Delta Lake as they were generally designed for relational databases where state is tracked through sequential DDL scripts.Silver LayerNew Columns - You can l...
- 0 kudos
- 145 Views
- 0 replies
- 0 kudos
Declarative Until It Isn't: Four Sharp Edges of Lakeflow Declarative Pipelines
The common case is almost too easy — which is exactly why the real engineering lives at the boundaries where the abstraction leaks.Declarative pipelines make the common case feel almost suspiciously easy. You describe the tables you want, point them ...
- 145 Views
- 0 replies
- 0 kudos
- 447 Views
- 1 replies
- 3 kudos
Resolved! Is it safe to expose JWT in Databricks Job?
There is an usecase where I want to verify/extract the job caller identity in the job runtime of databricks. The job run_as and creator parameter seems to be fixed at the job configuration and doesn't reflect/update who actually triggered a job run.A...
- 447 Views
- 1 replies
- 3 kudos
- 3 kudos
Hello !What I can say is that your conclusion is correct since DBKS does not currently expose the identity of the principal that called run-now to the running job through the Jobs API or notebook context and the audit event is the authoritative sourc...
- 3 kudos
- 349 Views
- 2 replies
- 1 kudos
Building a Visitor Data Pipeline for Digital Membership Card
Hi everyone, I'm working on a project where we collect visitor activity from multiple touchpoints such as ticketing, membership sign-ups, event participation, mobile app interactions, and digital membership card usage. The goal is to create a unified...
- 349 Views
- 2 replies
- 1 kudos
- 1 kudos
Concrete answer for the pieces you asked about, from a similar build (loyalty/membership 360 profile at bank scale - different domain, same shape of problem): Data model: model this as classic medallion, but the key decision is at silver/gold - use a...
- 1 kudos
- 246 Views
- 1 replies
- 0 kudos
Power BI Refresh Fails with "Unable to Continue Fetch After Reconnect" from Azure Databricks
Hi everyone,I'm refreshing a report with dataset (~10 million rows) from Azure Databricks in power Bi service. The refresh runs for about an hour and then fails with: Unable to continue fetch after reconnect. Retry limit exceeded. I am currently unab...
- 246 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi Raj,Query execution on Databricks completed in under a second but the failure is occurring during the client-side result streaming phase between Power BI and Azure Databricks.When importing Power BI streams result chunks over HTTP connections. If ...
- 0 kudos
- 395 Views
- 2 replies
- 2 kudos
Serverless Scala JAR: foreachBatch fails with RST_STREAM PROTOCOL_ERROR
Hi everyone,We're migrating a Scala Structured Streaming application (Kinesis → Delta) from classic compute to Databricks Serverless Jobs for cost optimization. During the migration, we're consistently seeing what appears to be a Spark Connect / gRPC...
- 395 Views
- 2 replies
- 2 kudos
- 2 kudos
On (1): the docs don't gate Scala foreachBatch on serverless. Limitations with Databricks Connect for Scala lists streaming foreachBatch as unavailable only on Databricks Runtime 13.3 LTS and below, and serverless limitations names only Trigger.Proce...
- 2 kudos
- 293 Views
- 0 replies
- 1 kudos
V.E.N.K.A.T Frameworkâ„¢
Hello Databricks Community,I recently published the V.E.N.K.A.T Frameworkâ„¢, an enterprise architecture model for the Agentic AI era.The idea is simple:As AI moves from generating insights to taking actions, the architecture underneath becomes critica...
- 293 Views
- 0 replies
- 1 kudos
- 2703 Views
- 5 replies
- 4 kudos
Resolved! Does VACUUM on Delta Lake also clean Iceberg metadata when using Iceberg Uniform feature?
I'm working with Delta tables using the Iceberg Uniform feature to enable Iceberg-compatible reads. I’m trying to understand how metadata cleanup works in this setup.Specifically, does the VACUUM operation—which removes old Delta Lake metadata based ...
- 2703 Views
- 5 replies
- 4 kudos
- 4 kudos
Sound discussion overall in this thread. Two hard earned lessons that came out of running this in production that I can think of sharing:(1) Rehearse the cleanup on a scratch table before relying on it: point your reader at a test table, expire snaps...
- 4 kudos
- 793 Views
- 5 replies
- 6 kudos
DLT pipeline cloning to another workspace.
I know about the clone API:<databricks-instance>/api/2.0/pipelines/<pipeline-id>/clonebut it only works within the same workspace.Is it possible to achieve the same result manually?As I understand:I can't choose where checkpoints are migrated to in U...
- 793 Views
- 5 replies
- 6 kudos
- 6 kudos
@yanchr To the checkpoint part of your question first. There is no supported way to carry DLT checkpoint state over to a different pipeline, and that includes a pipeline in another workspace. This is true even on hive metastore, where the pipeline's ...
- 6 kudos