- 106 Views
- 4 replies
- 0 kudos
Datavault implemention on Databricks
Hi everyone,I'm building a Data Vault 2.0 architecture entirely natively in Databricks from scratch. We have specifically decided not to use dbt.While Raw Vault ingestion is straightforward with Spark/Delta, I'm looking for architectural advice on co...
- 106 Views
- 4 replies
- 0 kudos
- 0 kudos
Great points from @balajij8 on the SDP/LDP approach. I've worked on a few Data Vault implementations, and these are the things that tend to bite when the Business Vault grows:1. Hash keys: standardize before hashing. Trim, upper-case and use a fixed ...
- 0 kudos
- 209 Views
- 2 replies
- 0 kudos
Resolved! Genie "Field" error: Table name or alias is required for column 'amount'
The documentation at https://docs.databricks.com/aws/en/genie-agents/tune-quality#define-sql-expressions says that you can create a "Field" in a genie agent (in the Examples section of config) with the code CASE WHEN amount < 10000 THEN 'Small' WHEN ...
- 209 Views
- 2 replies
- 0 kudos
- 0 kudos
Greetings @dkxxx-rc, I did some digging and here is what I found. Vannurswamy Kuruba already nailed the core of this, so I'll build on his reply. On #1: yes, it's a doc inconsistency. On the same page, the Filter example qualifies the column (orders....
- 0 kudos
- 620 Views
- 5 replies
- 3 kudos
DLT Pipeline - Overwrite except for one Append table
Hi all,I am a junior engineer and I am working on a use case involving DLT pipelines that read from a csv source at regular intervals and populate tables based upon processing of this. In most cases, I'd like the table contents to be overwritten with...
- 620 Views
- 5 replies
- 3 kudos
- 3 kudos
Hi. Please find the another update on the response below:Use @DP.materialized_view for the overwrite table and an append-only streaming table (@DP.table over an Auto Loader read) for the history table. I also need to retract something from my last r...
- 3 kudos
- 359 Views
- 1 replies
- 0 kudos
Declarative Until It Isn't: Four Sharp Edges of Lakeflow Declarative Pipelines
The common case is almost too easy — which is exactly why the real engineering lives at the boundaries where the abstraction leaks.Declarative pipelines make the common case feel almost suspiciously easy. You describe the tables you want, point them ...
- 359 Views
- 1 replies
- 0 kudos
- 0 kudos
Nice write-up, the read_files plus mergeSchema point is one I'd have missed. A few things from the current docs that may help with edges 2 and 4, since some of it has changed recently.On expectations: the incremental refresh page now says materialize...
- 0 kudos
- 1844 Views
- 0 replies
- 1 kudos
How to Unit Test Spark Declarative Pipelines
Every engineer who ships application code writes unit tests for it. Yet the data pipeline that feeds the same application is often left with a manual run and a dashboard that someone eventually notices is wrong. The gap has been more about tooling th...
- 1844 Views
- 0 replies
- 1 kudos
- 3627 Views
- 5 replies
- 2 kudos
Python logger.info() not showing inside applyInPandas (but print() works) — why?
Problem: In Databricks, logs from an external binary (via os.system) show up, but Python logger.info() inside groupBy(...).applyInPandas(...) does not. print(..., flush=True) does show up.Why: applyInPandas runs your function as a pandas UDF. That c...
- 3627 Views
- 5 replies
- 2 kudos
- 2 kudos
Hi Ben, The stdout logging-handler trick works on a classic/standard cluster, but on serverless it will not surface in the notebook cell, and that is expected rather than a problem with the helper. Here is what is going on. applyInPandas runs your Py...
- 2 kudos
- 4178 Views
- 8 replies
- 12 kudos
Future of Movie Discovery: How I Built an AI Movie Recommendation Agent on Databricks Free Edition
As a data engineer deeply passionate about how data and AI can come together to create real-world impact, I’m excited to share my project for the Databricks Free Edition Hackathon 2025 — Future of Movie Discovery (FMD). Built entirely on Databricks F...
- 4178 Views
- 8 replies
- 12 kudos
- 12 kudos
@Brahmareddy I haven't used this exact pattern at multi-million-row scale, but I agree that the tradeoffs start to change as the dataset and query volume grow.I would probably keep Delta as the source of truth and separate the serving layer based on ...
- 12 kudos
- 1686 Views
- 5 replies
- 0 kudos
Kill/Cancel a Notebook Cell Running Too Long on an All-purpose Cluster
Hi everyone, I’m facing an issue when running a notebook on a Databricks All-purpose cluster. Some of my cells/pipelines run for a very long time, and I want to automatically cancel/kill them when they exceed a certain time limit.I tried setting spar...
- 1686 Views
- 5 replies
- 0 kudos
- 0 kudos
Greetings @zenwanderer, I did some digging and here is what I found. Your follow-up narrows the ask: you want the workspace, not each developer, to catch and kill any interactive cell that runs past a threshold. Short answer: there's no supported, bu...
- 0 kudos
- 745 Views
- 5 replies
- 5 kudos
Resolved! Unity Catalog v.2.0?
Are there plans for a v.2.0 of Unity Catalog? I find the organization of tables in databricks to be very constrained and inflexible. Given that UC tables have their origins in data lakes, you would think they would have brought a lot more flexibile...
- 745 Views
- 5 replies
- 5 kudos
- 5 kudos
Fair points, especially for data coming from systems with richer naming. I can't speak to the roadmap, but here's what works for us today:Aliases: Create views in a consumer-facing catalog, like reporting.sales.outside_sales, that point to the real t...
- 5 kudos
- 365 Views
- 3 replies
- 3 kudos
Which Topics Deserve More Attention Before the Databricks Data Engineer Associate Certification Exam
As soon as I began my preparation for the Databricks Data Engineer Associate Certification exam, I came to the realisation that merely going through the syllabus won’t be sufficient. There are certain topics which are recurrent in practical situation...
- 365 Views
- 3 replies
- 3 kudos
- 3 kudos
Hey @AdamLucas I agree that just reading the syllabus isn’t enough for the Databricks Data Engineer Associate exam. I focused more on Delta Lake, Spark SQL, data transformation, and workflows because these topics needed a better understanding of how ...
- 3 kudos
- 755 Views
- 6 replies
- 3 kudos
Resolved! Power BI Dataflow Gen1 → Databricks Pipeline: Is There a Migration Accelerator or Converter?
Hello Great Analytics professionals:Many organizations have legacy Power BI Dataflow Gen1 solutions and are now considering moving their ETL workloads to Databricks.Is there an existing Databricks migration accelerator, converter, or framework that c...
- 755 Views
- 6 replies
- 3 kudos
- 3 kudos
@nafikazi There isn't a one-click Power BI Dataflow Gen1 / Power Query M to Lakeflow conversion path yet available.For a POC, decompose each flow and use Genie Code to accelerate the translation into Databricks SQL/ PySpark.Provide the M query, sourc...
- 3 kudos
- 599 Views
- 4 replies
- 2 kudos
Resolved! Unity Catalog Volume as spark checkpoint location (in 2026)
Hi folks,I'd like my checkpoint folder to be easily accessible to me in my UC volume files. This was possible in Fabric OneLake (Files). For whatever reason, it is not easy in databricks volumes. I keep getting meaningless errors: spark.sparkConte...
- 599 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi,Glad it helped. On 14.3, a few things to keep straight before you pick plan B.The error you got isn't about managed vs external. sc.setCheckpointDir hands the path to the JVM's local file API (that java.io.File canonicalize in the stack), and the ...
- 2 kudos
- 1455 Views
- 9 replies
- 2 kudos
Resolved! Using autoloader with multiple object types in load path
My data source is going to generate csv files for multiple objects all into the same directory that I need to load from. The files will have names in the format along the lines of <objecttype>_YYYY_MM_DD_guid.csv.gz. Each objecttype will have it's ow...
- 1455 Views
- 9 replies
- 2 kudos
- 2 kudos
A single Auto Loader stream with filename-based routing looks much cleaner here, especially since new object types can appear without updating the pipeline.
- 2 kudos
- 2327 Views
- 13 replies
- 2 kudos
Unable to Enable Unity Catalog – Azure Managed Identity Credential Not Found
Hi Databricks Community,I am trying to enable Unity Catalog in my Azure Databricks account. As part of the setup, I created an Azure Databricks Access Connector and assigned its managed identity the required permissions on my ADLS Gen2 storage accoun...
- 2327 Views
- 13 replies
- 2 kudos
- 2 kudos
I hit the same error.When creating a Unity Catalog metastore from the web UI of the Azure Databricks Account Console, the following error appeared during Access Connector validation, despite having all the Azure configuration correctly set up (manage...
- 2 kudos
- 516 Views
- 4 replies
- 0 kudos
AI/BI Dashboards
Hi AllMy client is looking to create a few dashboards in DataBricks instead of Power BI. We already have a template in Power BI (header for each page, information about out BI Page) in each of our power bi report which has colors, fonts, images etc.H...
- 516 Views
- 4 replies
- 0 kudos
- 0 kudos
Hi @km1837, Databricks AI/BI Dashboards have solid theming and organisational capabilities that should cover most of what you are used to in Power BI. AI/BI Dashboards don't have a "template file" concept the way Power BI does with .pbit files, but y...
- 0 kudos