cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

sudopradip
by • New Contributor II
  • 712 Views
  • 7 replies
  • 3 kudos

Serverless notebook: UC volume PUT via SQL connector fails *.storage.azuredatabricks.net resolves

From Azure Databricks serverless notebook compute (East US), uploading a file to a Unity Catalog volume with the Databricks SQL Connector PUT ... INTO '/Volumes/...' fails at TCP connect with [Errno 111] Connection refused. The warehouse accepts the ...

  • 712 Views
  • 7 replies
  • 3 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 3 kudos

Hello @sudopradip, I took a look at both internal and external documentation and here is what I found. Credit first. @SumeshKashyap had the practical answer early: write through /Volumes or the Files API from inside Databricks, don’t pin the public I...

  • 3 kudos
6 More Replies
EAnthemNHC1
by • New Contributor III
  • 3352 Views
  • 5 replies
  • 2 kudos

Time Travel Error when selecting from materialized view (Azure Databricks)

Hey - running into an error this morning that was brought to my attention via failed refreshes from PowerBI. We have a materialized view that, when queried with the standard pattern of 'select col1 from {schema}.table_name', returns an error of 'Cann...

  • 3352 Views
  • 5 replies
  • 2 kudos
Latest Reply
Alexjohn
New Contributor
  • 2 kudos

This is an interesting discussion about handling time travel errors when working with materialized views in Azure Databricks. Understanding how different data objects support historical queries is important, especially when troubleshooting complex da...

  • 2 kudos
4 More Replies
kapil_dua
by • Databricks Partner
  • 444 Views
  • 4 replies
  • 0 kudos

Data Ingestion Lakeflow UI Bug - Does not allow to create streaming table with BINARY data type

Data Ingestion (Lakeflow) Feature Bug:While creating Data Ingestion streaming table for ingesting pdf files, so that it can store into Binary format, it doesn't allow to select for BINARY data type for a column like (content), but it chose default da...

  • 444 Views
  • 4 replies
  • 0 kudos
Latest Reply
saurabh18cs
Honored Contributor III
  • 0 kudos

Hi @kapil_dua This works for me so what are you doing differently?  

  • 0 kudos
3 More Replies
NaniOnyx07
by • New Contributor
  • 274 Views
  • 2 replies
  • 2 kudos

Unable to Attach Databricks Serverless Compute – Environment Version 5 (Heavy Load Warning)

Hi Databricks Community,I'm experiencing an issue with Databricks Serverless Compute while trying to attach it to my notebook using Serverless Environment Version 5.Whenever I attempt to attach the compute, I receive the following warning: "Databrick...

Screenshot 2026-11-06 085609.png
  • 274 Views
  • 2 replies
  • 2 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 2 kudos

Hello @NaniOnyx07, I took a look at both internal and external documentation and here is what I found. First, credit to @ThomazNeto for the thorough first pass. I checked his facts against the docs and he is spot on: environment version 5 shipped in ...

  • 2 kudos
1 More Replies
faruk
by • Contributor II
  • 411 Views
  • 3 replies
  • 0 kudos

Resolved! Schema evolution: new NOT NULL column fails on existing Silver data (Lakeflow Connect source)

Hi everyone,We ingest data from our source database with Lakeflow Connect, through a foreign catalog, into a Bronze layer. In the Silver layer, we apply the correct data types and constraints based on our data dictionary.Recently, some tables in the ...

  • 411 Views
  • 3 replies
  • 0 kudos
Latest Reply
data_pulse
Contributor
  • 0 kudos

Good points already shared on the nullable → backfill → NOT NULL approach. I'd add a few Lakeflow Connect and Silver recovery considerations, particularly to avoid unnecessary full refreshes.1. Lakeflow Connect: Verify historical data before backfill...

  • 0 kudos
2 More Replies
faruk
by • Contributor II
  • 474 Views
  • 5 replies
  • 6 kudos

Resolved! How do you collect Excel files from business users into Databricks ?

Hi everyone,I build reports in Databricks from Excel files maintained by many different business users. The files are varied and come from different teams, and I haven’t settled on an approach yet.One idea I’m considering: create a landing location w...

  • 474 Views
  • 5 replies
  • 6 kudos
Latest Reply
balajij8
Esteemed Contributor III
  • 6 kudos

You can use a Unity Catalog Volume backed by S3 as the landing zone and its the right way. You can give each source team their own folder in the volume, allowing users to drop files through the UI. Pipeline can pick up files on arrival using a job fi...

  • 6 kudos
4 More Replies
jAAmes_bentley
by • Databricks Partner
  • 507 Views
  • 3 replies
  • 1 kudos

Resolved! Zerobus support for private-endpoint storage (MLflow traces to Unity Catalog)

We'd like to store MLflow 3 traces in Unity Catalog, using an experiment linked to a UC table-prefix trace location. In our Azure UK South workspace, linking the location fails with:PERMISSION_DENIED: Zerobus failed to open the table. Make sure the t...

  • 507 Views
  • 3 replies
  • 1 kudos
Latest Reply
LingeshK
Databricks Employee
  • 1 kudos

Hello @jAAmes_bentley  Following up on the two answers above with the doc lines behind them, one Azure change that affects your third question, and an option for now. Thank you for quoting the full error and stating the storage policy up front. That ...

  • 1 kudos
2 More Replies
ifra_inam
by • New Contributor II
  • 358 Views
  • 3 replies
  • 2 kudos

Databricks Free Edition: AWS S3 Unity Catalog Storage Credential Fails AssumeRole Validation

I'm using Databricks Free Edition and trying to configure an AWS S3 external location through Unity Catalog.My storage credential validation consistently fails at Assume Role, while Self Assume Role and External ID Condition are skipped.Error: Failed...

  • 358 Views
  • 3 replies
  • 2 kudos
Latest Reply
ShamenParis
Contributor III
  • 2 kudos

Hi @ifra_inam ,The PERMISSION_DENIED failure at the very first Assume Role stage means AWS IAM is instantly rejecting the cross-account trust handshake. Because the failure happens immediately, Databricks never even reaches the S3 permissions check.C...

  • 2 kudos
2 More Replies
turagittech
by • Contributor II
  • 518 Views
  • 3 replies
  • 3 kudos

Resolved! Update Ingestion pipleline to use pager duty with API

Hi,I have been working to update a cdc ingestion pipeline to add a notification to it on failure. We have had two instances where a maintenance task by vendor as tripped up CDC with some grief.I reviewed all the information and I am getting an error ...

Data Engineering
bundles
CDC
  • 518 Views
  • 3 replies
  • 3 kudos
Latest Reply
AbhilashNagilla
Databricks Employee
  • 3 kudos

Hi @turagittech, a pipeline's [notifications](https://docs.databricks.com/api/pipelines/v2/edit) setting accepts only email_recipients and alerts, while notification destinations such as PagerDuty are added in a job's notification settings. For the ...

  • 3 kudos
2 More Replies
abhietc31
by • New Contributor
  • 396 Views
  • 3 replies
  • 0 kudos

Real time Streaming to AI Engine at Databricks

Hi there,Usercase: Fraud Detection Technology : Databricks, Kafka, Microsoft Foundry, Dashboard showing real-time reportsAsk: Please let me know if any reference or technology details to achieve 1-2 milli second (Max) latency using above technology s...

  • 396 Views
  • 3 replies
  • 0 kudos
Latest Reply
AbhilashNagilla
Databricks Employee
  • 0 kudos

Hi @abhietc31, the fraud detection Solution Accelerator scores Kafka transactions in real-time mode and feeds a live Databricks Apps dashboard from Lakebase. For your own pipeline, the Lakebase sink writes real-time mode output to a Lakebase table wi...

  • 0 kudos
2 More Replies
Phani1
by • Databricks MVP
  • 1745 Views
  • 4 replies
  • 0 kudos

Resolved! Databricks Cost Estimation Template

Hi Databricks Team, Is there a standard Databricks cost estimation template(xl), sizing calculator, or TCO tool that allows us to provide the following inputs and derive an approximate monthly and annual platform cost:Source systems and their types (...

  • 1745 Views
  • 4 replies
  • 0 kudos
Latest Reply
Harshad_B
New Contributor III
  • 0 kudos

@Phani1 @robert7878 - You can build a AI/BI dashboard pretty easily using Genie, just give what is required and how you want it to be represented.

  • 0 kudos
3 More Replies
i4ata
by • New Contributor II
  • 533 Views
  • 3 replies
  • 1 kudos

Mismatch of spark.catalog.tableExists and SHOW TABLES in streams

Consider this minimal example of a streaming workflow:import os from pyspark.sql import functions as F, DataFrame from typing import Literal # Change these if you want to replicate CATALOG, SCHEMA = 'partner_demo_catalog', 'ad_sales_demo' spark.sql...

  • 533 Views
  • 3 replies
  • 1 kudos
Latest Reply
saurabh18cs
Honored Contributor III
  • 1 kudos

Hi @i4ata Hi  how if you try with batch specific session (just curious)def foreach_batch(batch_df, batch_id): spark_session = batch_df.sparkSession  if spark_session.catalog.tableExists(table_name): brSaurabh

  • 1 kudos
2 More Replies
Labels