- 117 Views
- 2 replies
- 0 kudos
Predictive Optimization for Streaming tables in Lakeflow pipelines
On July 2025 https://www.databricks.com/blog/whats-new-lakeflow-declarative-pipelines-july-2025 predictive optimization was enabled for all UC managed Lakeflow pipelines. I was wondering if there is a possibility now to disable it for Streaming table...
- 117 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Gilk, Before the how, can I ask what is driving the wish to turn it off? For most pipeline tables predictive optimization is doing useful work, running OPTIMIZE, VACUUM, and ANALYZE on serverless compute so file sizes, storage, and statistics sta...
- 0 kudos
- 204 Views
- 4 replies
- 0 kudos
Lakeflow connect CT pipeline keeps running multiple connections in source
I'm working on an ingestion pipeline. I was adding tables to it gradually so that we can monitor the load on our prod server side by side and with my last set of tables added (with them, all the heavy duty tables were inside that pipeline), the load ...
- 204 Views
- 4 replies
- 0 kudos
- 0 kudos
For a standard connector, the gateway continuously extracts snapshots, change logs, and metadata, while a separately scheduled serverless pipeline applies staged data (connector components). Changing the serverless pipeline schedule therefore does no...
- 0 kudos
- 121 Views
- 2 replies
- 0 kudos
Databricks Free Edition
Hi,I just signed up for Databricks free edition to start my learning journey in Data Engineering on it.Want to know for how long it remains free?Thank you.
- 121 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @kabiromohd, Welcome to Databricks, and a great choice for starting your Data Engineering journey! The short answer is that Databricks Free Edition is forever free. There is no trial period or expiration date. You can use it indefinitely as lo...
- 0 kudos
- 62 Views
- 0 replies
- 0 kudos
Managing "Secret" Injection in DABs across Dev/Stage/Prod
Hi everyone, I am fully moving our team to Databricks Asset Bundles (DABs) for CI/CD, but I’m struggling with the "Secret" management pattern.How are you handling the injection of secrets (like API keys for ingestion) into DABs without hardcoding any...
- 62 Views
- 0 replies
- 0 kudos
- 59 Views
- 0 replies
- 0 kudos
Architectural Pattern for PII: "Redact at Ingestion" vs. "Dynamic Masking at Consumption
Hi everyone, we are defining our PII strategy in Unity Catalog. We are split on whether to: A) Redact/Hash PII at the Silver layer (permanent change), or B) Keep PII in Silver and use Dynamic Data Masking at the Gold/View layer.Does "Redact at Inges...
- 59 Views
- 0 replies
- 0 kudos
- 63 Views
- 0 replies
- 0 kudos
Managing Row-Level Security (RLS) vs. Views: Performance impacts at scale
Hi everyone, we are designing our security layer in Unity Catalog. We are debating between using Row-Level Security (RLS) predicates versus standard SQL Views for masking PII data.When applying complex RLS predicates to tables with >100M rows, have ...
- 63 Views
- 0 replies
- 0 kudos
- 55 Views
- 0 replies
- 0 kudos
Scaling Delta Sharing across Non-Databricks Recipients: Handling Credential Rotation and Lifecycle
Hi everyone,Hi everyone, I’m architecting a data-sharing program using Delta Sharing for external partners who do not have a Databricks workspace.How are others automating the recipient credential lifecycle? Do you have an automated process for rotat...
- 55 Views
- 0 replies
- 0 kudos
- 944 Views
- 9 replies
- 2 kudos
Resolved! Using autoloader with multiple object types in load path
My data source is going to generate csv files for multiple objects all into the same directory that I need to load from. The files will have names in the format along the lines of <objecttype>_YYYY_MM_DD_guid.csv.gz. Each objecttype will have it's ow...
- 944 Views
- 9 replies
- 2 kudos
- 2 kudos
A single Auto Loader stream with filename-based routing looks much cleaner here, especially since new object types can appear without updating the pipeline.
- 2 kudos
- 137 Views
- 2 replies
- 2 kudos
Liquid Clustering vs Z-Order vs Partitioning
Liquid clustering: what it actually replaced, and the caveats worth knowingI see a lot of questions here that boil down to "should I partition, Z-order, or use liquid clustering." Posting how I reason about it, partly to help and partly because I'd ...
- 137 Views
- 2 replies
- 2 kudos
- 2 kudos
This lines up well with what Databricks' own docs and the 2026 refresh of their partitioning guidance say — a few things worth adding as confirmation/extra data points rather than pushback:On the key-count cap: confirmed at 4 clustering keys max. Wor...
- 2 kudos
- 173 Views
- 3 replies
- 3 kudos
Notebook commands hang after restarting a cluster with custom library installed
Hi,I'm experiencing an issue with a Databricks notebook after restarting a cluster that has a custom library installed.Here are the steps I followed:Created a Databricks compute cluster.Attached a notebook to the cluster.Installed a library required ...
- 173 Views
- 3 replies
- 3 kudos
- 3 kudos
Thank you for the response. From the driver logs I gather that the REPL has crashed, but this isn't the case only with Hive. I have noticed this pattern with Oracle library installation as well.
- 3 kudos
- 68 Views
- 0 replies
- 0 kudos
Data ingestion: Setting up a connector for Postgres with CDC enabled.
Hello,I'm trying to ingest data from my RDS instance and set up change data capture on Databricks. Everything on the Postgres side has been done so that replication is possible, but CDC is still greyed out. I have asked Claude, Gemini, and ChatGPT. I...
- 68 Views
- 0 replies
- 0 kudos
- 67 Views
- 0 replies
- 0 kudos
Best practice to enforcing row-level security across multiple catalogs sharing same schema structure
Hi Everyone,We have identical schema structures replicated across three catalogs (dev, qa, prod) to support environment isolation. We now need to apply row-level security (e.g., restricting sales reps to only see their own region's data) consistently...
- 67 Views
- 0 replies
- 0 kudos
- 602 Views
- 5 replies
- 4 kudos
Resolved! Issue in "Build a Declarative Pipeline with Spark Declarative Pipelines"
I am trying to complete the "Get Started with Data Engineering" course, but I have been running into an issue.I have gotten to this step in "Build a Declarative Pipeline with Spark Declarative Pipelines":Demo: Create and Run the PipelineNow you'll co...
- 602 Views
- 5 replies
- 4 kudos
- 4 kudos
I have added the source now. I went to the root folder > Create pipeline asset > New pipeline source code folder. I was then able to go to the right of the pipeline name at the top of the screen and edit the catalogue and schema.However, after runnin...
- 4 kudos
- 137 Views
- 3 replies
- 1 kudos
"Run now with different settings” is disabled for Continuous Jobs even when the trigger is paused.
Problem:When a Databricks Job is configured with a Continuous trigger, the “Run now with different settings” option remains disabled even after the Continuous trigger is paused.Limitation:This prevents us from manually running the same job with diffe...
- 137 Views
- 3 replies
- 1 kudos
- 7032 Views
- 7 replies
- 1 kudos
Connect Oracle Fusion (ERP / HCM) to Databricks
Any suggestion to connect Oracle Fusion (ERP/HCM) to Databricks?I have explored a few options including the use of Oracle Integration Cloud but it requires a lot of customization.
- 7032 Views
- 7 replies
- 1 kudos
- 1 kudos
Check out BI Connector(https://www.biconnector.com/oracle-fusion-data-warehouse-integration/), which offers direct connectivity from Oracle Fusion/FDI/OAC to DW/Lakehouse/Power BI/Tableau and more. It's widely deployed among Oracle Fusion & Analyti...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
