- 208 Views
- 11 replies
- 3 kudos
MS Fabric Direct Lake, works with delta tables only?
Lately we started using Direct Lake to use the results of our ingestion pipelines (gold tables) in PBI dashboards. This is working ok but has one annoying and troublesome issue. Which is: the gold tables are the end result of declarative pipelines so...
- 208 Views
- 11 replies
- 3 kudos
- 3 kudos
Hello dbernstein_tp, I took a look at both internal and external documentation and here is what I found. You're not missing anything. The Fabric Mirrored Azure Databricks Catalog item (including the newer Publish to OneLake button in Catalog Explorer...
- 3 kudos
- 111 Views
- 2 replies
- 0 kudos
Clarification on use_describe_as_json_for_relation_metadata and system.information_schema dependency
Hi Databricks Support Team,We are evaluating whether dbt can execute successfully without access to system.information_schema in our environment, due to metadata access restrictions.Our current setup uses:dbt-core: 1.12.3dbt-databricks: 1.12.5Configu...
- 111 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi @Aju_Antony, were you able to try the get_uc_tables_sql override in your POC? Quick way to confirm it works: run dbt run with --debug and check that no query in the log references system.information_schema.If that unblocked you, please click Accep...
- 0 kudos
- 652 Views
- 14 replies
- 9 kudos
Datavault implemention on Databricks
Hi everyone,I'm building a Data Vault 2.0 architecture entirely natively in Databricks from scratch. We have specifically decided not to use dbt.While Raw Vault ingestion is straightforward with Spark/Delta, I'm looking for architectural advice on co...
- 652 Views
- 14 replies
- 9 kudos
- 9 kudos
Hello @ankit2604, I took a look at both internal and external documentation and here is what I found. Thanks @balajij8, @Satyasai, @Jen_Evangelista and @data_pulse for doing most of the work here. The shape they've described is right: streaming table...
- 9 kudos
- 281 Views
- 6 replies
- 1 kudos
Table does not support batch scan.
After moving our incremental load jobs from DBR 16.4 (Spark 3.5) to DBR 18 (Spark 4.1.0), they fail with this error: [UNSUPPORTED_FEATURE.TABLE_OPERATION] The feature is not supported: Table `app_dev`.`vw_period` does not support batch scan. Please c...
- 281 Views
- 6 replies
- 1 kudos
- 1 kudos
Hello @harsh120, I took a look at both internal and external documentation and here is what I found. Building on @data_pulse and @Jen_Evangelista: RemoteSparkConnectScan is the key clue, and this doesn't look like MAP(), isEmpty(), or ANSI mode being...
- 1 kudos
- 1486 Views
- 18 replies
- 4 kudos
Change data feed from a materialized view
Hi everyone,Trying the CDF on materialized view (declarative pipeline) feature that is currently in Beta.- Configuration:** Verified the MVs (materialized veiws) have: 'delta.enableRowTracking'SHOW TBLPROPERTIES my_mv ('delta.enableRowTracking'); re...
- 1486 Views
- 18 replies
- 4 kudos
- 4 kudos
Hello @cdn_yyz_yul, Thanksto @Jen_Evangelista for the pipeline-config versus table-property framing and to @data_pulse for the Free Edition repro. On the property question: I can't point to a doc line, but the evidence says it's expected that pipelin...
- 4 kudos
- 74 Views
- 3 replies
- 2 kudos
Is Genie Ontology Available in Databricks Free Edition?
Hi Community,I wanted to confirm whether Genie Ontology is available in the Databricks Free Edition or if it is exclusive to paid editions.If it is supported in the Free Edition, could someone please guide me on how to access and enable this feature ...
- 74 Views
- 3 replies
- 2 kudos
- 2 kudos
@shubham_1507 Can validate this directly in Databricks Free Edition workspace.Open Genie One and ask:“Is Genie Ontology available in this workspace?” I have tested in Free edition workspace as below, In the expanded Thinking steps, Genie invoked sear...
- 2 kudos
- 48 Views
- 0 replies
- 0 kudos
Feature request: let DABs alert resources load query_text from an external .sql file
Hi everyone,We manage SQL alerts with Declarative Automation Bundles and like having them in version control. One thing we'd like to improve: the alert's SQL query has to be written inline in the YAML resource definition (query_text).The problemOur a...
- 48 Views
- 0 replies
- 0 kudos
- 614 Views
- 5 replies
- 2 kudos
Using Databricks Asset Bundles and Lakeflow Jobs in a Real Project
Hi everyone,I’m working through Databricks deployment and orchestration concepts and wanted to understand how Databricks Asset Bundles and Lakeflow Jobs fit together.Real-world scenario:Imagine an e-commerce company has a customer data pipeline that ...
- 614 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi @gowri_databrick In short, you use the bundle to define what gets deployed and where, and Lakeflow Jobs to control when it runs. (Asset Bundles are now called Declarative Automation Bundles, but the databricks bundle CLI is the same.) How bundles ...
- 2 kudos
- 10036 Views
- 13 replies
- 2 kudos
Seeking Guidance on Migrating Informatica PowerCenter Workflows to Databricks using Lakebridge
Hi everyone,I hope you're doing well.I'm currently exploring options to migrate a significant number of Informatica PowerCenter workflows and mappings to Databricks. During my research, I came across Lakebridge, especially its integration with BladeB...
- 10036 Views
- 13 replies
- 2 kudos
- 2 kudos
Hi,We've helped teams work through this exact decision (Synapse dedicated pool plus Spark ingestion, Fabric-only vs keeping Databricks). A few points that usually settle it:1. Split the decision by workload, not by platform.- T-SQL warehouse workload...
- 2 kudos
- 1918 Views
- 2 replies
- 1 kudos
Precision Variance Observed in FLOAT to DOUBLE Data Migration to Delta Tables
Hi Team,We would like to bring to your attention a precision-related variance observed during data migration from our legacy platform into db Delta tables.In the legacy system, several numeric columns are defined using the FLOAT data type. During ing...
- 1918 Views
- 2 replies
- 1 kudos
- 1 kudos
Hi @bunny_9090,Steve's answer covers the widening mechanics well. There are two things we check before choosing a fix on legacy-to-Delta migrations, because they can change the root cause completely.1. Confirm what "FLOAT" means in your source engine...
- 1 kudos
- 2188 Views
- 6 replies
- 1 kudos
Resolved! Design Oracle Fusion SCM to Azure Databricks
Hello Techie,I am planning to migrate All module of Oracle fusion scm data to Azure Databricks.Do we have only option of BICC (Business Intelligence Cloud Connector), OR any other option avaialble.Can anyone please help me with reference architecture...
- 2188 Views
- 6 replies
- 1 kudos
- 1 kudos
The BI Connector writes the Fusion PVO data straight to ADLS Gen2 or any lakehouse/warehouse platform of your choice. Please check out this resource for more info - https://www.biconnector.com/blog/oracle-bicc-integration/
- 1 kudos
- 166 Views
- 3 replies
- 5 kudos
Resolved! Excel Add-in Trouble (Confirm Button not Enabled)
Has anyone ever gotten the excel add-in to work on Azure Databricks, for Excel? I created a trusted catalog for the Excel Add-in, I launch the Excel, and activate the add-in, I sign into my workspace, I click the button to connect, I select my SQL wa...
- 166 Views
- 3 replies
- 5 kudos
- 5 kudos
hi @DB1To3 hope the answer was helpful, I would request you to accept it as solution so that it is useful for other community members
- 5 kudos
- 163 Views
- 2 replies
- 0 kudos
Feature Request: Cross-job dependencies with business-date awareness
SummaryWorkflows can chain tasks inside one job, but there is no native way to say: “Run job X once per day, after jobs A, B, C, D and E have each completed successfully for the same business date, even though they are independent jobs with their own...
- 163 Views
- 2 replies
- 0 kudos
- 0 kudos
Hi indradbxlearn, one edge case is worth making explicit in the gate: a successful run that started today is not necessarily a successful run for today’s business date. A backfill for an older date could satisfy a start_time_from=midnight test, while...
- 0 kudos
- 235 Views
- 3 replies
- 3 kudos
Resolved! Spark Event Logging to DBFS creates only 0-byte .inprogress file
Hi Everyone,I'm trying to capture Spark Event Logs for a Databricks cost optimization solution.I configured my job cluster as follows:spark.eventLog.enabled=truespark.eventLog.dir=dbfs:/databricks/spark-events/The cluster starts successfully and I ca...
- 235 Views
- 3 replies
- 3 kudos
- 3 kudos
Hi @Vamsi3757 - yes, the eventlog/ folder holds the full Spark listener event log, the same JSON-lines stream the Spark UI / History Server is built from: job, stage and task start/end events with task metrics, executor added/removed, SQL execution e...
- 3 kudos
- 375 Views
- 5 replies
- 2 kudos
Databricks Connect + AutoDependencies + UDF + Streaming breaks
I have this toy python script:CATALOG = 'mpt_usd_gbl_bronze_dev' # <-- replaceSCHEMA = 'mpt_gbl_tco_fof' # <-- replacesrc=f'{CATALOG}.{SCHEMA}.toy_source'DEST = f'{CATALOG}.{SCHEMA}.toy_dest'from databricks.connect import DatabricksSession, Databrick...
- 375 Views
- 5 replies
- 2 kudos
- 2 kudos
Thanks for sharing the detailed reproducer. It looks like separating the AutoDependencies hook error from the serverless UDF dependency limitation is key to troubleshooting this issue. Testing on compatible classic compute seems like a good next step...
- 2 kudos
| User | Count |
|---|---|
| 1645 | |
| 793 | |
| 586 | |
| 349 | |
| 287 |
