Databricks Community

It shouldn't. You have append only stream and SkipChangeCommit will ignore any modification that were applied to already existing files 

Hi @Han_bbb ,Could you provide more details? Is it workspace in your work or your private one? Which cloud provider? 

Reference Link - https://docs.databricks.com/aws/en/volumes/volume-files#upload-files-to-a-volume

I think there is no other way. In any case, here is how I usually configure my (all-purpose and jobs compute) clusters to access secrets via environment variables so that you don't have to update all references if some similar issue arises again. The...

This issue is related to how Delta Lake’s structured streaming interacts with schema evolution and options like startingVersion and schemaTrackingLocation. The behavior you've observed has been noted by other users, and can be subtle due to how check...

There is indeed a better and more integrated way to make Delta Lake tables accessible in Redshift without manually generating manifests and dynamically creating external tables or partitions. Some important points and options: Databricks Delta Lake ...

The most effective way to display joined data from Azure Databricks tables (like Employee and Salary) in a React JS UI involves exposing your Databricks data through an API and then consuming that API in your frontend. Flask can work, but there are b...

Auto Loader file notification in Databricks relies on Azure Event Grid’s BlobCreated event to trigger notifications for newly created files in Azure Data Lake Gen2. The issue you’re experiencing is a known limitation when files are written via certai...

Your challenge—reliably determining the subset of changed Databricks Asset Bundles after a Merge Request (MR) is merged into main for focused deploy/destroy CI/CD actions—is common in complex monorepo, multi-environment setups. Let’s break down the p...

Current Approach Assessment Power BI Import Mode: Importing all table data results in full dataset refreshes, driving up compute and data transfer costs during each refresh. Delta Lake as Source: Databricks clusters are used for both ETL and respon...

Yes, the speed difference you are seeing when querying Delta Shared tables versus local Delta tables is expected due to the architectural nature of Delta Sharing and network constraints. Why Delta Sharing Is Slower When you query a standard Delta tab...

The core answer is: Many users encounter failures in structured streaming pipelines when switching from Databricks normal (classic) compute to Serverless, especially when using read streams on Unity Catalog Delta tables with Change Data Feed (CDF) en...

When chaining stateful operators like groupBy (aggregation) and join in Spark Structured Streaming, there are specific rules about the output mode required for the overall query and the behavior of each operator. Output Mode Requirements The groupBy...

All good in @mark_ott response. As a potential improvement, instead of using polling, I think it would be better to publish events to a Bus (i.e. Azure Event Hub) from notebooks so that consumers could launch queries when receiving, processing and fi...