- 120 Views
- 3 replies
- 0 kudos
What is a Checkpoint in Structured Streaming?
Hi everyone,I’m learning about Structured Streaming in Databricks and came across checkpoints.I understand that checkpoints are used to keep track of the progress of a streaming query, but I’d like to understand their purpose more clearly.For example...
- 120 Views
- 3 replies
- 0 kudos
- 0 kudos
@gowri_databrick Yes, checkpoints are basically how Structured Streaming remembers where it got to.For example, imagine a streaming pipeline processing customer transactions:Source → Structured Streaming → Delta tableSuppose the pipeline has processe...
- 0 kudos
- 93 Views
- 3 replies
- 0 kudos
Understanding Parquet File Storage for Large Datasets
Hi everyone,I’m learning about Parquet files and how they are used in Databricks for storing large datasets.I’m trying to understand how column-based storage works in a practical situation.For example, suppose an e-commerce company has 500 million or...
- 93 Views
- 3 replies
- 0 kudos
- 0 kudos
@gowri_databrick Parquet's columnar storage organizes data by column rather than by row - all values for order_date are stored together, all order_amount values are stored together and so on. In your commerce scenario with 500 million records, when t...
- 0 kudos
- 334 Views
- 1 replies
- 2 kudos
why micro-batching matters so much in Databricks Auto Loader and Structured Streaming
I recently spent some time understanding why micro-batching matters so much in Databricks Auto Loader and Structured Streaming, and it changed the way I look at streaming pipelines.Earlier, I used to think streaming meant processing every record the ...
- 334 Views
- 1 replies
- 2 kudos
- 2 kudos
Absolutely agree. The key takeaway for me is that streaming doesn't automatically mean ultra-low latency. Choosing between processingTime and AvailableNow, and lower-latency approaches, should really come down to the business SLA, data arrival patter...
- 2 kudos
- 94 Views
- 1 replies
- 0 kudos
What is a Data Skipping in Delta Lake?
Hi everyone,I’m learning about Delta Lake performance and came across data skipping.I understand that it can help Databricks avoid reading unnecessary data when running queries, but I’d like to understand its purpose more clearly.For example, if an o...
- 94 Views
- 1 replies
- 0 kudos
- 0 kudos
@gowri_databrick Data skipping is a built in optimization that uses file-level statistics (minimum values, maximum values and null counts) to automatically skip reading data files that don't contain relevant data for your query. When you write data t...
- 0 kudos
- 138 Views
- 2 replies
- 1 kudos
Clarification on Automating Serverless Compute Permissions via SDK/API
Hi Team,I'm trying to automate granting Can Use permissions on Default Interactive Compute for workspace groups across multiple Databricks workspaces. While reviewing the Serverless Compute access control documentation and the Access Control Rule Set...
- 138 Views
- 2 replies
- 1 kudos
- 1 kudos
@saicharan1 you found a gap in the Azure documentationDatabricks creates Default Interactive Compute and Default Automated Compute objects in each workspace. Workspace admins manage their permissions, so you must grant CAN_USE in each workspace, even...
- 1 kudos
- 237 Views
- 0 replies
- 0 kudos
Who is The Best Cold Email Service Provider in 2026?
I have been working in the SMMA field for more than 5 years. So I have very strong experience about that.I tried many cold email providers; I use more than 10k emails from every provider, but everyone has a common problem. They will now give you a lo...
- 237 Views
- 0 replies
- 0 kudos
- 481 Views
- 3 replies
- 2 kudos
Serverless Scala JAR: foreachBatch fails with RST_STREAM PROTOCOL_ERROR
Hi everyone,We're migrating a Scala Structured Streaming application (Kinesis → Delta) from classic compute to Databricks Serverless Jobs for cost optimization. During the migration, we're consistently seeing what appears to be a Spark Connect / gRPC...
- 481 Views
- 3 replies
- 2 kudos
- 2 kudos
On (1): the docs don't gate Scala foreachBatch on serverless. Limitations with Databricks Connect for Scala lists streaming foreachBatch as unavailable only on Databricks Runtime 13.3 LTS and below, and serverless limitations names only Trigger.Proce...
- 2 kudos
- 256 Views
- 3 replies
- 1 kudos
Resolved! How can i rename a column in a delta table?
I have a delta table and i want to rename one of its columns.what is the recommended way to rename a column in databricks?is there any difference between renaming a column using sql and using pyspark?
- 256 Views
- 3 replies
- 1 kudos
- 1 kudos
@gowri_databrick Common Approaches are:SQL migration through PySpark: This is only a Python wrapper around the same SQL DDL. It is not a different renaming mechanism. It needs the columnMapping.mode = 'name' to be enabled on the table first.spark.sql...
- 1 kudos
- 242 Views
- 1 replies
- 1 kudos
cross-region DR in Azure Databricks (24h RPO/RTO)
Hi,We're designing a DR strategy for an Azure Databricks platform and would appreciate guidance on current best practices for achieving approximately 24-hour RPO and RTO across Azure regions.Our platform includes Unity Catalog, DAB, Jobs, Notebooks, ...
- 242 Views
- 1 replies
- 1 kudos
- 1 kudos
Your overall design is sensible, but Databricks’ current recommendation is to use Managed Disaster Recovery when your account is eligible. If Managed DR is unavailable, use an active-passive warm standby with Terraform/DABs, cross-region data replica...
- 1 kudos
- 525 Views
- 2 replies
- 2 kudos
Resolved! databricks SQL UDF in select statement
In the Unity Catalog we can now create/register SQL UDFs. There are two types - one that returns table and other that returns just a value. If the function that returns value is based on the SQL query and joins it would in standard relational databas...
- 525 Views
- 2 replies
- 2 kudos
- 2 kudos
The earlier reply has the mechanism broadly right, and your two questions have clean answers: yes to the first, no to the second. A SQL scalar function whose body is a query is planned as a scalar subquery inside the calling statement. Databricks lab...
- 2 kudos
- 511 Views
- 3 replies
- 0 kudos
Resolved! Unexpected behavior of Delta VACUUM – need explanation
HelloI have a question about how VACUUM works.We executed the following VACUUM command from a PySpark job:spark.sql("""VACUUM catalog_name.schema_name.table_name RETAIN 168 HOURS""")The Delta table has a table property retention period of 365 days, b...
- 511 Views
- 3 replies
- 0 kudos
- 0 kudos
Greetings @bricks_2026 , glad the first answer helped. Your sliding window observation is sharp, but one correction to the mental model: delta.deletedFileRetentionDuration only drives physical deletion at the moment a VACUUM executes. Between your w...
- 0 kudos
- 448 Views
- 3 replies
- 0 kudos
How to extract data from SAP to Databricks?
Hey everyone, I'm working through a project where I need to get data out of SAP S4Hana and into Databricks, and I'd love to pick the brains of anyone who's done this before.A few things I'm trying to figure out:How are you actually pulling the data o...
- 448 Views
- 3 replies
- 0 kudos
- 0 kudos
Hi,SAP BDC is one option to consider.SAP Databricks in Business Data Cloud sap-bdc OpenSharing
- 0 kudos
- 594 Views
- 2 replies
- 0 kudos
Resolved! Error DELTA_CATALOG_MANAGED_TABLE_UPGRADE_WITH_OTHER_PROPERTIES during catalog commit upgrade
Hi everyone,I'm encountering an issue with the Catalog Commit functionality while attempting to upgrade a table, following the official Databricks documentationCatalog commits | Databricks on AWSWhen running the following command on a Managed Iceberg...
- 594 Views
- 2 replies
- 0 kudos
- 0 kudos
This looks like an interaction between the catalog-managed protocol upgrade and VARIANT shredding, rather than an issue with the SQL syntax itself.delta.feature.catalogManaged needs to be enabled as a standalone table/protocol upgrade. In this case, ...
- 0 kudos
- 470 Views
- 3 replies
- 0 kudos
How should schema evolution be handled across silver and gold layers in a medallion architecture?
We run a medallion pipeline on Databricks:Bronze: AutoLoader ingests raw CSV files into Delta tables (append-only, all columns as STRING, schema evolution via addNewColumns)Silver: PySpark jobs clean and transform bronze data into Delta tables using ...
- 470 Views
- 3 replies
- 0 kudos
- 0 kudos
You can skip using traditional schema migration tools like Liquibase, Flyway or Alembic into Delta Lake as they were generally designed for relational databases where state is tracked through sequential DDL scripts.Silver LayerNew Columns - You can l...
- 0 kudos
- 176 Views
- 0 replies
- 0 kudos
Declarative Until It Isn't: Four Sharp Edges of Lakeflow Declarative Pipelines
The common case is almost too easy — which is exactly why the real engineering lives at the boundaries where the abstraction leaks.Declarative pipelines make the common case feel almost suspiciously easy. You describe the tables you want, point them ...
- 176 Views
- 0 replies
- 0 kudos