- 906 Views
- 9 replies
- 2 kudos
Resolved! Using autoloader with multiple object types in load path
My data source is going to generate csv files for multiple objects all into the same directory that I need to load from. The files will have names in the format along the lines of <objecttype>_YYYY_MM_DD_guid.csv.gz. Each objecttype will have it's ow...
- 906 Views
- 9 replies
- 2 kudos
- 2 kudos
A single Auto Loader stream with filename-based routing looks much cleaner here, especially since new object types can appear without updating the pipeline.
- 2 kudos
- 107 Views
- 2 replies
- 2 kudos
Liquid Clustering vs Z-Order vs Partitioning
Liquid clustering: what it actually replaced, and the caveats worth knowingI see a lot of questions here that boil down to "should I partition, Z-order, or use liquid clustering." Posting how I reason about it, partly to help and partly because I'd ...
- 107 Views
- 2 replies
- 2 kudos
- 2 kudos
This lines up well with what Databricks' own docs and the 2026 refresh of their partitioning guidance say — a few things worth adding as confirmation/extra data points rather than pushback:On the key-count cap: confirmed at 4 clustering keys max. Wor...
- 2 kudos
- 149 Views
- 3 replies
- 3 kudos
Notebook commands hang after restarting a cluster with custom library installed
Hi,I'm experiencing an issue with a Databricks notebook after restarting a cluster that has a custom library installed.Here are the steps I followed:Created a Databricks compute cluster.Attached a notebook to the cluster.Installed a library required ...
- 149 Views
- 3 replies
- 3 kudos
- 3 kudos
Thank you for the response. From the driver logs I gather that the REPL has crashed, but this isn't the case only with Hive. I have noticed this pattern with Oracle library installation as well.
- 3 kudos
- 56 Views
- 0 replies
- 0 kudos
Data ingestion: Setting up a connector for Postgres with CDC enabled.
Hello,I'm trying to ingest data from my RDS instance and set up change data capture on Databricks. Everything on the Postgres side has been done so that replication is possible, but CDC is still greyed out. I have asked Claude, Gemini, and ChatGPT. I...
- 56 Views
- 0 replies
- 0 kudos
- 55 Views
- 0 replies
- 0 kudos
Best practice to enforcing row-level security across multiple catalogs sharing same schema structure
Hi Everyone,We have identical schema structures replicated across three catalogs (dev, qa, prod) to support environment isolation. We now need to apply row-level security (e.g., restricting sales reps to only see their own region's data) consistently...
- 55 Views
- 0 replies
- 0 kudos
- 536 Views
- 5 replies
- 4 kudos
Resolved! Issue in "Build a Declarative Pipeline with Spark Declarative Pipelines"
I am trying to complete the "Get Started with Data Engineering" course, but I have been running into an issue.I have gotten to this step in "Build a Declarative Pipeline with Spark Declarative Pipelines":Demo: Create and Run the PipelineNow you'll co...
- 536 Views
- 5 replies
- 4 kudos
- 4 kudos
I have added the source now. I went to the root folder > Create pipeline asset > New pipeline source code folder. I was then able to go to the right of the pipeline name at the top of the screen and edit the catalogue and schema.However, after runnin...
- 4 kudos
- 128 Views
- 3 replies
- 1 kudos
"Run now with different settings” is disabled for Continuous Jobs even when the trigger is paused.
Problem:When a Databricks Job is configured with a Continuous trigger, the “Run now with different settings” option remains disabled even after the Continuous trigger is paused.Limitation:This prevents us from manually running the same job with diffe...
- 128 Views
- 3 replies
- 1 kudos
- 7027 Views
- 7 replies
- 1 kudos
Connect Oracle Fusion (ERP / HCM) to Databricks
Any suggestion to connect Oracle Fusion (ERP/HCM) to Databricks?I have explored a few options including the use of Oracle Integration Cloud but it requires a lot of customization.
- 7027 Views
- 7 replies
- 1 kudos
- 1 kudos
Check out BI Connector(https://www.biconnector.com/oracle-fusion-data-warehouse-integration/), which offers direct connectivity from Oracle Fusion/FDI/OAC to DW/Lakehouse/Power BI/Tableau and more. It's widely deployed among Oracle Fusion & Analyti...
- 1 kudos
- 177 Views
- 5 replies
- 4 kudos
using remote_query with SQL Serverless Warehouse
Hello, Im trying to set up remote_query() as an option for our developers to use when querying our DB2 server. Unfortunately the connection keeps timing out. The workspace is in a VPC in AWS and the interactive clusters can successfully use the JDBC ...
- 177 Views
- 5 replies
- 4 kudos
- 4 kudos
A few things worth separating out here, because the symptom (timeout, not a handshake/certificate error) points away from your working theory:Question the diagnosis first: if the TLS handshake itself were the problem — no way to present/validate a ce...
- 4 kudos
- 3488 Views
- 5 replies
- 1 kudos
Deleting Records from DLT Bronze and Silver Tables
I have a pipeline that generates two DLT streaming tables: a Bronze table and a Silver table. I need to delete specific records from both tables. I've read an article (https://www.databricks.com/blog/handling-right-be-forgotten-gdpr-and-ccpa-using-de...
- 3488 Views
- 5 replies
- 1 kudos
- 1 kudos
The core idea: the reason deleting from Bronze breaks the Silver streaming read isn't Delta itself — it's that Structured Streaming (which DLT streaming tables use under the hood) requires an append-only source by default. Any DELETE/UPDATE on Bronze...
- 1 kudos
- 146 Views
- 2 replies
- 0 kudos
Training Notebooks/ Access to Lab
Hi, I am looking for training notebooks or access to lab
- 146 Views
- 2 replies
- 0 kudos
- 0 kudos
Also can check Databricks Academy Labs https://www.databricks.com/databricks-academy-labsDb Demos Tutorials: https://www.databricks.com/resources/demos/tutorials Db Demos cover various use cases by just doing below in any databricks Workspace. Try th...
- 0 kudos
- 33923 Views
- 3 replies
- 0 kudos
What are the best practices for optimizing Power BI reports and dashboards for performance in the PL
I’m studying for the PL-300 exam and would love some advice on how to optimize Power BI reports and dashboards for better performance. Specifically, I’m interested in:Techniques for improving report load times and responsiveness.Best practices for ma...
- 33923 Views
- 3 replies
- 0 kudos
- 0 kudos
A few practices can make a significant difference when optimizing Power BI reports and dashboards for performance, especially when Power BI is connected to Databricks or other large data sources.1. Optimize the data model firstUse a simple star schem...
- 0 kudos
- 189 Views
- 3 replies
- 2 kudos
DLT Pipeline - Overwrite except for one Append table
Hi all,I am a junior engineer and I am working on a use case involving DLT pipelines that read from a csv source at regular intervals and populate tables based upon processing of this. In most cases, I'd like the table contents to be overwritten with...
- 189 Views
- 3 replies
- 2 kudos
- 2 kudos
If history means changed versions by key, define Table1 as a batch materialized view and feed it to AUTO CDC FROM SNAPSHOT; its source can be a table or view, so the CSV doesn't need streaming semantics (Python datasets, snapshot API). from pyspark i...
- 2 kudos
- 716 Views
- 7 replies
- 12 kudos
create_auto_cdc_from_snapshot_flow Python session resolution fails if having multiple snapshot flows
When a pipeline contains more than one create_auto_cdc_from_snapshot_flow flow (each driven by a custom Python next_snapshot_and_version function), flow resolution fails intermittently/consistently with: RuntimeError: The original Spark session is be...
- 716 Views
- 7 replies
- 12 kudos
- 12 kudos
Hi again, it seems to have regressed on me, my pipeline is failing again, anybody else seeing this?
- 12 kudos
- 294 Views
- 6 replies
- 10 kudos
Resolved! Photon enabled but a large share of the plan is falling back, cost up and runtime flat
Hi everyone,Trying to work out whether this is expected or whether I have misconfigured something.We enabled Photon on a job cluster running a nightly aggregation over roughly 2TB. The expectation was the usual improvement. What we got instead was ru...
- 294 Views
- 6 replies
- 10 kudos
- 10 kudos
Great questions! Would love to hear from someone who’s dealt with Photon fallbacks, especially around Python UDFs. Curious to know if rewriting them actually made a noticeable difference in runtime and cost.
- 10 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 584 | |
| 349 | |
| 287 |
