- 97 Views
- 3 replies
- 3 kudos
Resolved! Excel Add-in Trouble (Confirm Button not Enabled)
Has anyone ever gotten the excel add-in to work on Azure Databricks, for Excel? I created a trusted catalog for the Excel Add-in, I launch the Excel, and activate the add-in, I sign into my workspace, I click the button to connect, I select my SQL wa...
- 97 Views
- 3 replies
- 3 kudos
- 3 kudos
hi @DB1To3 hope the answer was helpful, I would request you to accept it as solution so that it is useful for other community members
- 3 kudos
- 69 Views
- 0 replies
- 0 kudos
MS Fabric Direct Lake, works with delta tables only?
Lately we started using Direct Lake to use the results of our ingestion pipelines (gold tables) in PBI dashboards. This is working ok but has one annoying and troublesome issue. Which is: the gold tables are the end result of declarative pipelines so...
- 69 Views
- 0 replies
- 0 kudos
- 74 Views
- 1 replies
- 0 kudos
Clarification on use_describe_as_json_for_relation_metadata and system.information_schema dependency
Hi Databricks Support Team,We are evaluating whether dbt can execute successfully without access to system.information_schema in our environment, due to metadata access restrictions.Our current setup uses:dbt-core: 1.12.3dbt-databricks: 1.12.5Configu...
- 74 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Aju_Antony Interesting question!!! Short answer: no flag removes it. In dbt-databricks 1.12.x, relation discovery for Unity Catalog is hardcoded to system.information_schema.tables (the get_uc_tables macro), and use_describe_as_json_for_relation_...
- 0 kudos
- 1407 Views
- 17 replies
- 3 kudos
Change data feed from a materialized view
Hi everyone,Trying the CDF on materialized view (declarative pipeline) feature that is currently in Beta.- Configuration:** Verified the MVs (materialized veiws) have: 'delta.enableRowTracking'SHOW TBLPROPERTIES my_mv ('delta.enableRowTracking'); re...
- 1407 Views
- 17 replies
- 3 kudos
- 3 kudos
I tried reproducing a similar scenario using serverless compute in Databricks Free Edition.I tested enabling pipelines.externalMetadata.enabled=true at the pipeline level, explicitly setting it in the materialized view's TBLPROPERTIES, and also creat...
- 3 kudos
- 559 Views
- 9 replies
- 11 kudos
Datavault implemention on Databricks
Hi everyone,I'm building a Data Vault 2.0 architecture entirely natively in Databricks from scratch. We have specifically decided not to use dbt.While Raw Vault ingestion is straightforward with Spark/Delta, I'm looking for architectural advice on co...
- 559 Views
- 9 replies
- 11 kudos
- 11 kudos
how to handle duplicates(arriving at different time and not in the same batch) when data is ingested from stage/bronze to hubs? LDP/DLT does not support merge operations and what if i have multiple source table ingesting into a single hub. Any though...
- 11 kudos
- 142 Views
- 2 replies
- 0 kudos
Feature Request: Cross-job dependencies with business-date awareness
SummaryWorkflows can chain tasks inside one job, but there is no native way to say: “Run job X once per day, after jobs A, B, C, D and E have each completed successfully for the same business date, even though they are independent jobs with their own...
- 142 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi indradbxlearn, one edge case is worth making explicit in the gate: a successful run that started today is not necessarily a successful run for today’s business date. A backfill for an older date could satisfy a start_time_from=midnight test, while...
- 0 kudos
- 197 Views
- 3 replies
- 3 kudos
Resolved! Spark Event Logging to DBFS creates only 0-byte .inprogress file
Hi Everyone,I'm trying to capture Spark Event Logs for a Databricks cost optimization solution.I configured my job cluster as follows:spark.eventLog.enabled=truespark.eventLog.dir=dbfs:/databricks/spark-events/The cluster starts successfully and I ca...
- 197 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Vamsi3757 - yes, the eventlog/ folder holds the full Spark listener event log, the same JSON-lines stream the Spark UI / History Server is built from: job, stage and task start/end events with task metrics, executor added/removed, SQL execution e...
- 3 kudos
- 228 Views
- 4 replies
- 0 kudos
Table does not support batch scan.
After moving our incremental load jobs from DBR 16.4 (Spark 3.5) to DBR 18 (Spark 4.1.0), they fail with this error: [UNSUPPORTED_FEATURE.TABLE_OPERATION] The feature is not supported: Table `app_dev`.`vw_period` does not support batch scan. Please c...
- 228 Views
- 4 replies
- 0 kudos
- 0 kudos
I also asked Databricks Genie about it, and it described this as a likely bug. I know its answers are not reliable, but it suggests the behavior is unexpected.For now we dowgraded to DBR 16.4
- 0 kudos
- 353 Views
- 5 replies
- 2 kudos
Databricks Connect + AutoDependencies + UDF + Streaming breaks
I have this toy python script:CATALOG = 'mpt_usd_gbl_bronze_dev' # <-- replaceSCHEMA = 'mpt_gbl_tco_fof' # <-- replacesrc=f'{CATALOG}.{SCHEMA}.toy_source'DEST = f'{CATALOG}.{SCHEMA}.toy_dest'from databricks.connect import DatabricksSession, Databrick...
- 353 Views
- 5 replies
- 2 kudos
- 2 kudos
Thanks for sharing the detailed reproducer. It looks like separating the AutoDependencies hook error from the serverless UDF dependency limitation is key to troubleshooting this issue. Testing on compatible classic compute seems like a good next step...
- 2 kudos
- 383 Views
- 4 replies
- 1 kudos
Resolved! Zerobus Variant Data not Processed as Expected (json data destined for Variant column).
Lets say I have a table receiving zerobus data (json) over HTTP/rest.That last VARIANT is "ValueJson".If I want to submit data to it, I would expect to be able to use normal json formatting:[ { "a": "E", "b": 123, "c": "xy...
- 383 Views
- 4 replies
- 1 kudos
- 1 kudos
Agree it feels unnatural, but you can hide it from the remote client with a thin encoding step right before sending, so producers keep working with normal nested JSON.A small helper that encodes only the VARIANT columns:import jsonVARIANT_COLUMNS = {...
- 1 kudos
- 292 Views
- 4 replies
- 0 kudos
About Azure SQL Managed Connector
Hi Community,We are trying to configure a Databricks Lakeflow Connect managed SQL Server connector to ingest data from Azure SQL Database into Databricks.We are using a contained SQL database user that has access only to the specific source database,...
- 292 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi srikanthp24, one distinction in the proposed workaround matters: permission to connect across databases and permission to read business tables are different security boundaries.The current Azure Databricks privilege matrix explicitly requires mast...
- 0 kudos
- 427 Views
- 1 replies
- 1 kudos
from_utc_timestamp silently double-shifts time when session timezone isn't UTC — docs should call th
Summary:from_utc_timestamp(current_timestamp(), '<tz>') produces an incorrect (future-shifted) timestamp whenever the Spark session timezone is already set to something other than UTC. This is a very common pattern for teams stamping dp_load_ts/creat...
- 427 Views
- 1 replies
- 1 kudos
- 1 kudos
Hi rikkyvai, the audit-timestamp warning is useful. One nuance: with TIMESTAMP_LTZ, current_timestamp() represents an instant; the session timezone changes its display. from_utc_timestamp() shifts the value, rather than just formatting it.A useful ch...
- 1 kudos
- 766 Views
- 12 replies
- 9 kudos
Resolved! Zerobus Ingest with Kafka compliant API and WITHOUT a callback
Is see that kafka for zerobus is still in beta. So maybe something will change. Can someone tell me whether they are aware of an authentication handshake with this API that might AVOID the callback?Here is the python example, which uses a sophistica...
- 766 Views
- 12 replies
- 9 kudos
- 9 kudos
I don’t see a way to skip the callback inside the Kafka API itself, but there are workarounds.What the docs say (page updated Oct 1, 2026):OAUTHBEARER is the only auth mechanism. There is no documented anonymous or unauthenticated mode, and I wouldn’...
- 9 kudos
- 2161 Views
- 5 replies
- 1 kudos
Resolved! Design Oracle Fusion SCM to Azure Databricks
Hello Techie,I am planning to migrate All module of Oracle fusion scm data to Azure Databricks.Do we have only option of BICC (Business Intelligence Cloud Connector), OR any other option avaialble.Can anyone please help me with reference architecture...
- 2161 Views
- 5 replies
- 1 kudos
- 1 kudos
Hi,We currently have a Fusion-to-Data Warehouse architecture using ADW, but we're facing some challenges around cost and performance. We're exploring an alternative approach to remove ADW, ingest the OIC PVO CSV extracts directly into Databricks, and...
- 1 kudos
- 957 Views
- 4 replies
- 1 kudos
How to Track Hourly or Daily # of Upsert/Delete Metrics in a DLT Streaming Pipeline
We created a Delta Live Tables (DLT) streaming pipeline to ingest data from the Bronze layer to the Silver layer using Change Data Feed (CDF) enabled.The stream runs continuously and shows # of upserted and deleted rows at an aggregate level from the...
- 957 Views
- 4 replies
- 1 kudos
- 1 kudos
Adding to the good suggestions already shared around DLT event logs and CDF, a couple of newer Databricks capabilities may help operationalise this monitoring.1. Lakeflow Pipeline Events system table (Beta)Databricks now provides a centralised system...
- 1 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
