- 725 Views
- 3 replies
- 4 kudos
Autoscaling with the autoloader without SDP
Hi there,I have a question regarding the autoloader without SDP and auto-scaling of clusters. I'm reading the following in the docs:Production considerations for Structured Streaming | Databricks on AWS:Do not enable autoscaling for compute for Struc...
- 725 Views
- 3 replies
- 4 kudos
- 4 kudos
Hi, thank you for your answer. Could you elaborate a bit on this?for non SDP available now auto loader jobs autoscaling can be reasonableHow do you decide on whether it is reasonable or not? Especially you said it is not recommended to enable compute...
- 4 kudos
- 1378 Views
- 2 replies
- 3 kudos
Connecting DBeaver to Databricks Lakebase — Setup & Troubleshooting
I recently connected DBeaver to Databricks Lakebase and wanted to share the setup steps along with a couple of troubleshooting issues I encountered.Since Lakebase is PostgreSQL-compatible, the standard PostgreSQL driver works directly without requiri...
- 1378 Views
- 2 replies
- 3 kudos
- 3 kudos
Hello @Abhishek_sinha ! Thanks for sharing this ! very useful Few things I can add (from my personal XP), it is better to use the PostgreSQL driver and not the DBKS JDBC driver because Lakebase is PostgreSQL compatible so DBeaver should be configur...
- 3 kudos
- 2937 Views
- 3 replies
- 1 kudos
Using Databrick Bladebridge or Lakebridge for SQL Migration
Getting Transpile Error while executing command for Databrick Bladebridge or Lakebridge,databricks labs lakebridge transpile --source-dialect mssql --input-source "<Path>/sample.sql" --output-folder "<Path>\output"Error :TranspileError(code=FAILURE, ...
- 2937 Views
- 3 replies
- 1 kudos
- 1 kudos
Looks like some dependency/runtime issue with LakeBridge installation on Windows.pip uninstall databricks-labs-lakebridge -ypip install databricks-labs-lakebridgeAlso check Python version compatibility once. Python 3.10/3.11 worked better in my case.
- 1 kudos
- 846 Views
- 1 replies
- 0 kudos
Lakeflow Connect Data ingestion from SQL Server and PostgreSQL to Databricks with CDC
We have a requirement to use Lakeflow Connect for data ingestion from SQL Server and PostgreSQL into Databricks with CDC and Lakehouse federation. I would like to understand the pros and cons of Lakeflow Connect in the following areas Firewall/gatewa...
- 846 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @shan-databricks You should setup postgresql for ingestion via Lakeflow connect. Once your Postgres logical replication is ready you have to create ingestion pipelines which comprise a gateway and ingestion pipeline. Your gateway pipeline is conti...
- 0 kudos
- 1366 Views
- 6 replies
- 4 kudos
Resolved! Lakebridge reconciliation code keeps running continuously without Spark jobs or errors
Hi,I am facing an issue while running the Lakebridge reconciliation code in Databricks using TriggerReconService.trigger_recon().The code keeps running continuously without any output, error, or logs. Also, no Spark jobs are getting triggered or show...
- 1366 Views
- 6 replies
- 4 kudos
- 4 kudos
Could you give an example of the config files you have set up for running the reconciliation? The file determines most of the settings - so without it it is hard to assist you
- 4 kudos
- 1311 Views
- 2 replies
- 1 kudos
Performance optimization on auto_cdc_flow
I've got a fact streaming table, which is updated by SCD2 records from the CDF of a silver table. The join is on pk (hash key generated from dimensions business keys) and factory_code (60 unique values). On each incremental processing, it reads all ...
- 1311 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @yit337 !This is expected for AUTO CDC with SCD2 and it is not doing a simple append because it must upsert incoming CDC rows into the target based on the declared keys and for SCD2 it also maintains historical rows with __START_AT or __END_AT.So...
- 1 kudos
- 2206 Views
- 4 replies
- 2 kudos
create_auto_cdc_from_snapshot_flow vs create_auto_cdc_flow – when is snapshot CDC actually worth it?
I am deciding between create_auto_cdc_from_snapshot_flow() and create_auto_cdc_flow() in a pipeline.My source is a daily full snapshot table:No operation column (no insert/update/delete flags)Order can be derived from snapshot_date (sequence by)Rows ...
- 2206 Views
- 4 replies
- 2 kudos
- 2 kudos
Does this work only for tables with PK. What if the source table doesnt even have PK. Does it use any type of hashing by concatenating all columns and then use that key for merge?
- 2 kudos
- 806 Views
- 2 replies
- 2 kudos
Resolved! Does Lakeflow Connect guarantee no out-of-order records?
I use Lakeflow Connect to load data from my source databases to bronze tables. Then I have auto_cdc to track SCD2 changes in my silver tables. I use _commit_timestamp from the bronze CDF, as sequence_by property in auto_cdc in order to order the vers...
- 806 Views
- 2 replies
- 2 kudos
- 2 kudos
Recommendation: use a business/effective timestamp in sequence_by if your source can emit late/backdated changes and you want SCD2 history to reflect source event time, not bronze arrival/commit time. If ties are possible, use a STRUCT for determinis...
- 2 kudos
- 624 Views
- 3 replies
- 1 kudos
Need to fetch Mount Point details
Hi Team,I’m currently working on building a consolidated view of access permissions across our Databricks environment.For Unity Catalog (UC) objects, I’m able to retrieve permission details using system tables (privileges / audit logs).However, for l...
- 624 Views
- 3 replies
- 1 kudos
- 1 kudos
Hello @Danish11052000 !Thank you for the question it really helped me to review my knowledge and go back and pay attention to this subject and guess what ? you are correct because UC permissions alone will not give complete access governance for leg...
- 1 kudos
- 2682 Views
- 4 replies
- 0 kudos
Resolved! Authentication failure Lakeflow SQL Server Ingestion
Hi all I am trying to create a Lakeflow Ingestion Pipeline for SQL Server, but I am running into the following authentication error when using my Databricks Database User for the connection:Gateway is stopping. Authentication failure while obtaining ...
- 2682 Views
- 4 replies
- 0 kudos
- 0 kudos
I am also facing the same issue. We have user id as service principal name however in sql log it shows applicationID that it cannot login . Setvice principal ( name) has highest privileges in sql db . howevrr same is working fine through jdbc
- 0 kudos
- 988 Views
- 2 replies
- 5 kudos
Resolved! Unity Catalog - How to read prod data in dev with appropriate read-only access?
Hi there,Our team is currently migrating to using Unity Catalog. We have two databricks workspaces for dev & prod, and one thing that I'm wondering is if there is a simple/appropriate way to have only two catalogs dev & prod, where the prod databrick...
- 988 Views
- 2 replies
- 5 kudos
- 5 kudos
Yes — you can accomplish exactly what you described with only two catalogs (dev + prod). You do not need a third prod_readonly catalog.There are two complementary control planes in Unity Catalog:Workspace-level restriction (workspace-catalog binding)...
- 5 kudos
- 1991 Views
- 6 replies
- 0 kudos
Trying to convert oracle sql to databricks sql but not getting the desired output
ORACLE SQL: COUNT( CASE WHEN TRUNC(WORKORDER.REPORTDATE) BETWEEN SELECTED_PERIOD_START_DATE AND SELECTED_PERIOD_END_DATE THEN 1 END ) SELECTED_PERIOD_BM,COUNT( CASE WHEN TRUNC(WORKORDER.REPORTDATE) BETWEEN COMPARISON_PERIOD_START_DATE AND COMPARISON_...
- 1991 Views
- 6 replies
- 0 kudos
- 0 kudos
You’re using date_format(...) which turns dates into strings, so BETWEEN becomes a string comparison. You can also look up for databricks lakebridge that can assist you in code conversion or migrations. https://databrickslabs.github.io/lakebridge/
- 0 kudos
- 1351 Views
- 4 replies
- 3 kudos
Resolved! AI/BI Dashboard refresh via DABs + Jobs executes successfully but dashboard does not update without
I’m migrating a solution from an on-prem setup to Databricks AI/BI Dashboards, and I’m trying to replicate a near real-time dashboard experience (around ~1 minute latency is acceptable).In the legacy setup, we used DirectQuery combined with automatic...
- 1351 Views
- 4 replies
- 3 kudos
- 3 kudos
Thanks for the answer! Thats unfortunate. Do you think in the future Databricks will support the provided use case or will we need to do workarounds such as embedding the Dashboard in an DBKS app or maybe just create the entire Dashboard in an app us...
- 3 kudos
- 999 Views
- 1 replies
- 1 kudos
Calls to databricks api taking more than 60 seconds to complete
Hi,since April 1st (2026) we are having problems calling the databricks /api/2.2/jobs/runs/list and the /api/2.0/sql/history/queries endpoint. Calls to these endpoints sometimes seem to take longer than 60 seconds now using the databricks python sdk ...
- 999 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @MiriamHundemer !I don't think this is a rate limit issue because it is indeed 30 req/sec per workspace but that only tells you when some throttling may happen and it does not guarantee that every call returns within 60 sec. Also don't forget ...
- 1 kudos
- 1014 Views
- 3 replies
- 3 kudos
Resolved! Declarative Automation Bundle - Reusable job_cluster configuration
Hi there, running into some trouble abstracting job_clusters configurations to improve reusability. At the moment, I have many job yaml files that require the following configuration:What would be the best approach(es) to remove this configuration fr...
- 1014 Views
- 3 replies
- 3 kudos
- 3 kudos
Hello @ChristianRRL My doubt about your issue is happening in cluster_definitions.yml because it is not only defining a reusable cluster profile it is also redefining the same jobs that already exist in the individual fleet_*.yml files.Why ? because ...
- 3 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 582 | |
| 349 | |
| 287 |
