- 305 Views
- 7 replies
- 9 kudos
How to Connect Databricks to Snowflake & Query Live Data?
Hi everyone,I am looking for a method to connect Databricks to Snowflake so I can query and transform the data directly in a notebook. I specifically want to avoid methods that just perform a one-time copy of the tables into Databricks, as I need the...
- 305 Views
- 7 replies
- 9 kudos
- 9 kudos
Hello @Raj_DB, I took a look at both internal and external documentation and here is what I found. The thread already has the right answer: Lakehouse Federation. @saurabh18cs, @Harshad_B, @AbhilashNagilla, @Aravind_Reddy, @Jen_Evangelista and @balaji...
- 9 kudos
- 471 Views
- 3 replies
- 1 kudos
Mismatch of spark.catalog.tableExists and SHOW TABLES in streams
Consider this minimal example of a streaming workflow:import os from pyspark.sql import functions as F, DataFrame from typing import Literal # Change these if you want to replicate CATALOG, SCHEMA = 'partner_demo_catalog', 'ad_sales_demo' spark.sql...
- 471 Views
- 3 replies
- 1 kudos
- 1 kudos
Hi @i4ata Hi how if you try with batch specific session (just curious)def foreach_batch(batch_df, batch_id): spark_session = batch_df.sparkSession if spark_session.catalog.tableExists(table_name): brSaurabh
- 1 kudos
- 110 Views
- 1 replies
- 0 kudos
Managed table conversion and vacuum
Hello everyone,I am looking for clarification about the 14-day rollback period after converting an external table to a managed table using: ALTER TABLE <table_name> SET MANAGEDDatabricks document...
- 110 Views
- 1 replies
- 0 kudos
- 0 kudos
Hi @Davide, this thread never got a reply, but you asked the same thing in SET MANAGED and rollback in relation to VACUUM and posted your test result there, so I'm linking it for anyone who lands here first.Short version: the 14-day rollback window i...
- 0 kudos
- 1096 Views
- 5 replies
- 0 kudos
How to extract data from SAP to Databricks?
Hey everyone, I'm working through a project where I need to get data out of SAP S4Hana and into Databricks, and I'd love to pick the brains of anyone who's done this before.A few things I'm trying to figure out:How are you actually pulling the data o...
- 1096 Views
- 5 replies
- 0 kudos
- 3057 Views
- 3 replies
- 7 kudos
How to Unit Test Spark Declarative Pipelines
Every engineer who ships application code writes unit tests for it. Yet the data pipeline that feeds the same application is often left with a manual run and a dashboard that someone eventually notices is wrong. The gap has been more about tooling th...
- 3057 Views
- 3 replies
- 7 kudos
- 7 kudos
Thanks for this, the quickstart was a great starting point. The per-test isolation with the redirecting_* schemas is really nice: we could test AUTO CDC edge cases (deletes, out-of-order updates, null keys) on mock data, without touching real tables....
- 7 kudos
- 2992 Views
- 15 replies
- 2 kudos
Unable to Enable Unity Catalog – Azure Managed Identity Credential Not Found
Hi Databricks Community,I am trying to enable Unity Catalog in my Azure Databricks account. As part of the setup, I created an Azure Databricks Access Connector and assigned its managed identity the required permissions on my ADLS Gen2 storage accoun...
- 2992 Views
- 15 replies
- 2 kudos
- 2 kudos
i have created access connector,storage account and databricks resource in same region.Reader and Storage Actions Blob Data Operator access are given.still getting below error Azure Managed Identity Credential with Access Connector Id /subscriptions/...
- 2 kudos
- 1138 Views
- 6 replies
- 7 kudos
Resolved! Unity Catalog v.2.0?
Are there plans for a v.2.0 of Unity Catalog? I find the organization of tables in databricks to be very constrained and inflexible. Given that UC tables have their origins in data lakes, you would think they would have brought a lot more flexibile...
- 1138 Views
- 6 replies
- 7 kudos
- 7 kudos
Fair points, especially for data coming from systems with richer naming. I can't speak to the roadmap, but here's what works for us today:Aliases: Create views in a consumer-facing catalog, like reporting.sales.outside_sales, that point to the real t...
- 7 kudos
- 1502 Views
- 14 replies
- 12 kudos
Resolved! Datavault implemention on Databricks
Hi everyone,I'm building a Data Vault 2.0 architecture entirely natively in Databricks from scratch. We have specifically decided not to use dbt.While Raw Vault ingestion is straightforward with Spark/Delta, I'm looking for architectural advice on co...
- 1502 Views
- 14 replies
- 12 kudos
- 12 kudos
Hello @ankit2604, I took a look at both internal and external documentation and here is what I found. Thanks @balajij8, @Satyasai, @Jen_Evangelista and @data_pulse for doing most of the work here. The shape they've described is right: streaming table...
- 12 kudos
- 555 Views
- 3 replies
- 5 kudos
Resolved! Excel Add-in Trouble (Confirm Button not Enabled)
Has anyone ever gotten the excel add-in to work on Azure Databricks, for Excel? I created a trusted catalog for the Excel Add-in, I launch the Excel, and activate the add-in, I sign into my workspace, I click the button to connect, I select my SQL wa...
- 555 Views
- 3 replies
- 5 kudos
- 5 kudos
hi @DB1To3 hope the answer was helpful, I would request you to accept it as solution so that it is useful for other community members
- 5 kudos
- 470 Views
- 2 replies
- 0 kudos
Resolved! Genie "Field" error: Table name or alias is required for column 'amount'
The documentation at https://docs.databricks.com/aws/en/genie-agents/tune-quality#define-sql-expressions says that you can create a "Field" in a genie agent (in the Examples section of config) with the code CASE WHEN amount < 10000 THEN 'Small' WHEN ...
- 470 Views
- 2 replies
- 0 kudos
- 0 kudos
Greetings @dkxxx-rc, I did some digging and here is what I found. Vannurswamy Kuruba already nailed the core of this, so I'll build on his reply. On #1: yes, it's a doc inconsistency. On the same page, the Filter example qualifies the column (orders....
- 0 kudos
- 788 Views
- 5 replies
- 3 kudos
DLT Pipeline - Overwrite except for one Append table
Hi all,I am a junior engineer and I am working on a use case involving DLT pipelines that read from a csv source at regular intervals and populate tables based upon processing of this. In most cases, I'd like the table contents to be overwritten with...
- 788 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi. Please find the another update on the response below:Use @DP.materialized_view for the overwrite table and an append-only streaming table (@DP.table over an Auto Loader read) for the history table. I also need to retract something from my last r...
- 3 kudos
- 474 Views
- 1 replies
- 0 kudos
Declarative Until It Isn't: Four Sharp Edges of Lakeflow Declarative Pipelines
The common case is almost too easy — which is exactly why the real engineering lives at the boundaries where the abstraction leaks.Declarative pipelines make the common case feel almost suspiciously easy. You describe the tables you want, point them ...
- 474 Views
- 1 replies
- 0 kudos
- 0 kudos
Nice write-up, the read_files plus mergeSchema point is one I'd have missed. A few things from the current docs that may help with edges 2 and 4, since some of it has changed recently.On expectations: the incremental refresh page now says materialize...
- 0 kudos
- 3799 Views
- 5 replies
- 2 kudos
Python logger.info() not showing inside applyInPandas (but print() works) — why?
Problem: In Databricks, logs from an external binary (via os.system) show up, but Python logger.info() inside groupBy(...).applyInPandas(...) does not. print(..., flush=True) does show up.Why: applyInPandas runs your function as a pandas UDF. That c...
- 3799 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi Ben, The stdout logging-handler trick works on a classic/standard cluster, but on serverless it will not surface in the notebook cell, and that is expected rather than a problem with the helper. Here is what is going on. applyInPandas runs your Py...
- 2 kudos
- 4493 Views
- 8 replies
- 12 kudos
Future of Movie Discovery: How I Built an AI Movie Recommendation Agent on Databricks Free Edition
As a data engineer deeply passionate about how data and AI can come together to create real-world impact, I’m excited to share my project for the Databricks Free Edition Hackathon 2025 — Future of Movie Discovery (FMD). Built entirely on Databricks F...
- 4493 Views
- 8 replies
- 12 kudos
- 12 kudos
@Brahmareddy I haven't used this exact pattern at multi-million-row scale, but I agree that the tradeoffs start to change as the dataset and query volume grow.I would probably keep Delta as the source of truth and separate the serving layer based on ...
- 12 kudos
- 1840 Views
- 5 replies
- 0 kudos
Kill/Cancel a Notebook Cell Running Too Long on an All-purpose Cluster
Hi everyone, I’m facing an issue when running a notebook on a Databricks All-purpose cluster. Some of my cells/pipelines run for a very long time, and I want to automatically cancel/kill them when they exceed a certain time limit.I tried setting spar...
- 1840 Views
- 5 replies
- 0 kudos
- 0 kudos
Greetings @zenwanderer, I did some digging and here is what I found. Your follow-up narrows the ask: you want the workspace, not each developer, to catch and kill any interactive cell that runs past a threshold. Short answer: there's no supported, bu...
- 0 kudos