- 170 Views
- 1 replies
- 1 kudos
Synced table pipeline fails with permission denied for database
Environment: Azure Databricks, Lakebase Autoscaling project. Database created through the Lakebase UI by the project owner (audit log shows createProjectBranchDatabaseUI with a browser user agent). Synced tables created via the databricks_postgres_sy...
- 170 Views
- 1 replies
- 1 kudos
- 1 kudos
Hello @KhaturiabPreet !I like you analysis it made me do some reading this morning to freshen up my memory.I checked the the doc and I can say that databricks_writer_<dbid> being unable to CONNECT does not look like an expected state because it is t...
- 1 kudos
- 11626 Views
- 9 replies
- 2 kudos
Resolved! Trouble Enabling File Events For An External Location
Hello all,I am trying to enable file events on my Azure Workspace for the File Arrival Trigger trigger mode for Databricks Workflows. I'm following this documentation exactly (I think) but I'm not seeing the option to enable them. As you can see here...
- 11626 Views
- 9 replies
- 2 kudos
- 2 kudos
I am also facing some issue with the "file arrival" trigger on one of my environments.Its strange that it kept running fine until today where it broke. I always only had the "Storage Blob Data Contributor" for my Managed Identity ( Access Connector) ...
- 2 kudos
- 246 Views
- 2 replies
- 1 kudos
Building a Visitor Data Pipeline for Digital Membership Card
Hi everyone, I'm working on a project where we collect visitor activity from multiple touchpoints such as ticketing, membership sign-ups, event participation, mobile app interactions, and digital membership card usage. The goal is to create a unified...
- 246 Views
- 2 replies
- 1 kudos
- 1 kudos
Concrete answer for the pieces you asked about, from a similar build (loyalty/membership 360 profile at bank scale - different domain, same shape of problem): Data model: model this as classic medallion, but the key decision is at silver/gold - use a...
- 1 kudos
- 279 Views
- 2 replies
- 2 kudos
Serverless Scala JAR: foreachBatch fails with RST_STREAM PROTOCOL_ERROR
Hi everyone,We're migrating a Scala Structured Streaming application (Kinesis → Delta) from classic compute to Databricks Serverless Jobs for cost optimization. During the migration, we're consistently seeing what appears to be a Spark Connect / gRPC...
- 279 Views
- 2 replies
- 2 kudos
- 2 kudos
On (1): the docs don't gate Scala foreachBatch on serverless. Limitations with Databricks Connect for Scala lists streaming foreachBatch as unavailable only on Databricks Runtime 13.3 LTS and below, and serverless limitations names only Trigger.Proce...
- 2 kudos
- 6626 Views
- 6 replies
- 8 kudos
[UNRESOLVED_ROUTINE] Cannot resolve function `date_format`
Hi all,We are getting the following error log in a Workflow:AnalysisException: [UNRESOLVED_ROUTINE] Cannot resolve function `date_format` on search path [`system`.`builtin`, `system`.`session`]. SQLSTATE: 42883The Workflow consists in different noteb...
- 6626 Views
- 6 replies
- 8 kudos
- 8 kudos
Having a similar issue when moving from DBR 13 to DBR 16.4 in a multi-task Job. The first task works ok but the 2nd with identical code but different params fails. For me it fails with Cannot resolve function `and` (I assume from the .between(from_da...
- 8 kudos
- 373 Views
- 2 replies
- 1 kudos
Streaming tables vs materialized views in declarative pipelines: what actually decides the choice?
I am trying to settle a design default for our declarative pipelines, and I would like to hear how others decide between streaming tables and materialized views when a table could be built either way.When I started with streaming tables, I thought th...
- 373 Views
- 2 replies
- 1 kudos
- 1 kudos
Good points, Brahmareddy — and I think we mostly agree, with one nuance worth teasing out on the cross-team part.A published streaming table already *is* a stable Delta table from the reader's perspective — consumers query it by its UC name and never...
- 1 kudos
- 322 Views
- 2 replies
- 2 kudos
DLT driver GC pressure during a large initial hydration. Is a bigger driver the only lever?
Last week one of our DLT pipelines spent five hours in RUNNING with nothing landing, and I'd like to compare notes on how people handle the underlying problem, because we found one fix and several dead ends.Setup: A continuous DLT pipeline on classic...
- 322 Views
- 2 replies
- 2 kudos
- 2 kudos
I ran into something similar, and the event log was the only place that made the root cause obvious. I don't believe there's an officially supported setting to limit concurrent DLT flows, so I'd avoid undocumented configs. For large initial backfills...
- 2 kudos
- 5688 Views
- 11 replies
- 3 kudos
Databricks Job cluster for continuous run
Hi AllI am having situation where I wanted to run job as continuous trigger by using job cluster, cluster terminating and re-creating in every run within continuous trigger.I just wanted two know if we have any option where I can use same job cluster...
- 5688 Views
- 11 replies
- 3 kudos
- 3 kudos
@Ajay-Pandey Are you trying to solve the cold start issue, or having continuously running cluster driven by business goals? It would be helpful to get some context here.
- 3 kudos
- 925 Views
- 4 replies
- 2 kudos
Resolved! sdp-meta (dlt-meta) vs lakeflow_framework: when should we use which?
Hi All,I'm evaluating metadata-driven frameworks for Lakeflow Spark Declarative Pipelines, and I've found two that appear to serve the same purpose, both from Databricks organizations:- databrickslabs/dlt-meta, whose feature/sdp-meta branch renames t...
- 925 Views
- 4 replies
- 2 kudos
- 2 kudos
Since you are already on Lakeflow Connect for MySQL, your CDC ingestion into Bronze is the managed, supported path, so you do not need a metadata framework for it. Resources by layer: Ingestion to Bronze (what you have): stay on Lakeflow Connect. It...
- 2 kudos
- 721 Views
- 4 replies
- 3 kudos
Resolved! How do you keep a plain Delta copy of a streaming table? CLONE is not supported on streaming tables
Hi All, I need an up-to-date plain Delta copy of a streaming table created by Lakeflow pipeline. My first thought was CLONE, and that path is closed. The clone docs say streaming tables and materialized views cannot be the source or target of a deep ...
- 721 Views
- 4 replies
- 3 kudos
- 3 kudos
Thanks @adnan_alvee , yes, we did test Compatibility Mode. The output is a copy of files at a storage location, not a Unity Catalog table.Context: We are setting up an external engine (duckdb) to read UC Delta tables through credential vending, which...
- 3 kudos
- 197 Views
- 1 replies
- 0 kudos
Ingesting data from a SharePoint Excel file with two worksheets in one pipeline
Hi,I would like to ask about ingesting data from an Excel file stored in SharePoint.Currently, I have one Excel file containing two worksheets:one worksheet contains dimension data,the other worksheet contains target values.I would like to ingest bot...
- 197 Views
- 1 replies
- 0 kudos
- 0 kudos
Based on the behavior you described, this looks like a Managed SharePoint ingestion bug or preview limitation, not an obvious error in your configuration.Databricks’ Excel reader supports selecting one worksheet with dataAddress, and your successful ...
- 0 kudos
- 929 Views
- 6 replies
- 4 kudos
Serverless egress - public internet access issues
I've just setup a new Databricks account on AWS with a workspace with "Serverless compute and default storage" and I am unable to access the public internet from notebooks and jobs for all but a handful of domains.A subset works fine, I've identified...
- 929 Views
- 6 replies
- 4 kudos
- 4 kudos
Hi there, from your note and comments, I couldn't gather if you have tried standard compute. Did you experience the same issues with standard compute? With serverless, you might experience some snags since the infrastructure is owned by Databricks an...
- 4 kudos
- 330 Views
- 1 replies
- 0 kudos
Dashboard "Export as HTML" fails for job-associated notebook dashboards (UI only)
Context:A notebook is scheduled to run as a Databricks Job. The notebook has an associated dashboard. When viewing the dashboard from the job run, exporting via "Export as HTML" fails.Error:When clicking "Export as HTML," the download fails and the f...
- 330 Views
- 1 replies
- 0 kudos
- 0 kudos
Greetings @amrim , this is a well-scoped report, and your isolation table does most of the diagnostic work for us. Thanks for laying it out that way. The behavior you documented is consistent, and it points at the product, not your notebook or cluste...
- 0 kudos
- 268 Views
- 1 replies
- 0 kudos
SCIM API shows only databricks-sql-access,but Workspace Access is still displayed (grayed out) in UI
Hi Team,I'm testing Databricks entitlement automation using the Workspace Groups SCIM API.I have an Entra ID group that is synchronized to the Databricks Account via SCIM. I'm using a Service Principal authentication to manage workspace-level entitle...
- 268 Views
- 1 replies
- 0 kudos
- 0 kudos
What you're seeing is expected, and it comes down to how "Workspace access" is granted right now versus how the other entitlements work. Right now, every workspace principal automatically inherits the "Workspace access" entitlement through im...
- 0 kudos
- 555 Views
- 1 replies
- 1 kudos
Resolved! Lakebase CDF — destination Delta table not created after successful UI setup (Free Edition)
Summary:We configured Lakebase CDF to stream changes from a native Postgres table (public.query_logs) into Unity Catalog, but the destination Delta table never appears despite completing all documented prerequisites.Steps taken1. Created query_logs ...
- 555 Views
- 1 replies
- 1 kudos
- 1 kudos
Most likely this is a Free Edition storage limitation rather than a bug, and your Postgres-side setup sounds correct. Lakebase CDF writes its destination as a Unity Catalog managed Delta table, and the Lakebase CDF docs list catalogs backed by defaul...
- 1 kudos