cancel
Showing results for 
Search instead for 
Did you mean: 
Databricks Platform Discussions
Dive into comprehensive discussions covering various aspects of the Databricks platform. Join the conversation to deepen your understanding and maximize your usage of the Databricks platform.
cancel
Showing results for 
Search instead for 
Did you mean: 

Browse the Community

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies with...

11923 Posts

Data Governance

Join discussions on data governance practices, compliance, and security within the Databricks Commun...

515 Posts

Generative AI

Explore discussions on generative artificial intelligence techniques and applications within the Dat...

344 Posts

Machine Learning

Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithm...

1004 Posts

Warehousing & Analytics

Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Communi...

667 Posts

Activity in Databricks Platform Discussions

prajwalpoojary
by > Visitor
  • 24 Views
  • 1 replies
  • 0 kudos

Databricks Apps Hosting Backend and Frontend

Hello, I want to host a webapp whose frontend will be on Streamlit and backend running on FastApi. Currently Databricks app listens to host 0.0.0.0 and port 8000 and my backend is running on host '127.0.0.1' and port 8080(if it's available). I want t...

  • 24 Views
  • 1 replies
  • 0 kudos
Latest Reply
stbjelcevic
Databricks Employee
  • 0 kudos

Hi @prajwalpoojary , Given you already have Streamlit on 0.0.0.0:8000 and FastAPI on 127.0.0.1:8080, you can keep that split and do server-side calls from Streamlit to http://127.0.0.1:8080/. It’s efficient and avoids cross-origin/auth issues. If you...

  • 0 kudos
APJESK
by > Contributor
  • 53 Views
  • 2 replies
  • 0 kudos

Workspace Folder ACL design

How should the Databricks workspace folder architecture be designed to support cross-team collaboration, access governance, and scalability in an enterprise platform? Please suggest below or share some ideas from your experience ThanksNote: I'm new t...

  • 53 Views
  • 2 replies
  • 0 kudos
Latest Reply
APJESK
Contributor
  • 0 kudos

Thanks for the detailed information, Iwill review and get back to you if any question meanwhile can you please on this query Databricks Workspace ACL Enforcement – How to Prevent Users from Creating Objects Outside Team Folder and Attaching to Shared...

  • 0 kudos
1 More Replies
4Twannie
by > New Contributor III
  • 581 Views
  • 5 replies
  • 3 kudos

Delta Sharing from Databricks to SAP BDC fails with invalid_client error

ContextWe are in the process of extracting data between SAP BDC Datasphere and Databricks (Brownfield Implementation).SAP Datasphere is hosted in AWS (eu10)Databricks is hosted in Azure (West Europe)The BDC Connect System is located in the same regio...

  • 581 Views
  • 5 replies
  • 3 kudos
Latest Reply
anshu_roy
Databricks Employee
  • 3 kudos

The error DELTA_SHARING_INVALID_RECIPIENT_AUTH refers to an invalid authorization specification when accessing Delta Sharing resources. This maps to SQLSTATE code 28000 ("invalid authorization specification") and typically occurs when the recipient's...

  • 3 kudos
4 More Replies
fgeriksen
by > Visitor
  • 55 Views
  • 2 replies
  • 0 kudos

Enabling External Lineage on a free or trial account?

Hi,as part of a small OSS project I am doing, dbt-unity-lineage, I need to enable Bring your own data lineage (Public Preview as of December 2025). But it seems you can't enable that Preview in either free edition or Trial?I'd rather not use my emplo...

Administration & Architecture
dbt
dbt GA PublicPreview
  • 55 Views
  • 2 replies
  • 0 kudos
Latest Reply
fgeriksen
Visitor
  • 0 kudos

Thanks, the trial is currently created as a Premium, did not see any options to choose otherwise. I tried use east and eu central, thinking it might have been a regional thing.But thanks for checking it out, and your reply.

  • 0 kudos
1 More Replies
andrefilipemm
by > Visitor
  • 26 Views
  • 0 replies
  • 0 kudos

Automatic Identity Management

Hello,In the context of reviewing our company's databricks structure and migrating legacy workspaces to Unity Catalog enabled ones, we're stuck with a few questions regarding enabling the automatic identity management feature.We currently provision D...

  • 26 Views
  • 0 replies
  • 0 kudos
Michael_Appiah
by > Contributor II
  • 15262 Views
  • 16 replies
  • 11 kudos

Parameterized spark.sql() not working

Spark 3.4 introduced parameterized SQL queries and Databricks also discussed this new functionality in a recent blog post (https://www.databricks.com/blog/parameterized-queries-pyspark)Problem: I cannot run any of the examples provided in the PySpark...

Michael_Appiah_0-1704459542967.png Michael_Appiah_1-1704459570498.png
  • 15262 Views
  • 16 replies
  • 11 kudos
Latest Reply
Malthe
Contributor III
  • 11 kudos

@adriennn this has nothing to do with DLT, but about Databricks providing a different session implementation here than regular Spark.

  • 11 kudos
15 More Replies
dave_d
by > New Contributor II
  • 9217 Views
  • 3 replies
  • 0 kudos

What is the "Columnar To Row" node in this simple Databricks SQL query profile?

I am running a relatively simple SQL query that writes back to a table on a Databricks serverless SQL warehouse, and I'm trying to understand why there is a "Columnar To Row" node in the query profile that is consuming the vast majority of the time s...

dave_d_0-1696974904324.png
  • 9217 Views
  • 3 replies
  • 0 kudos
Latest Reply
Annapurna_Hiriy
Databricks Employee
  • 0 kudos

 @dave_d We do not have a document with list of operations that would bring up ColumnarToRow node. This node provides a common executor to translate an RDD of ColumnarBatch into an RDD of InternalRow. This is inserted whenever such a transition is de...

  • 0 kudos
2 More Replies
liquibricks
by > Contributor
  • 389 Views
  • 8 replies
  • 4 kudos

Declarative Pipeline error: Name 'kdf' is not defined. Did you mean: 'sdf'

We have a Lakeflow Spark Declarative Pipeline using the new PySpark Pipelines API. This was working fine until about 7am (Central European) this morning when the pipeline started failing with a PYTHON.NAME_ERROR: name 'kdf' is not defined. Did you me...

  • 389 Views
  • 8 replies
  • 4 kudos
Latest Reply
liquibricks
Contributor
  • 4 kudos

It turns out this problem was caused by a package that was pip installed using an init script. This package had for some reason started pulling in pandas 3.x (despite the fact that the package itself had not been updated), and our Databricks contact ...

  • 4 kudos
7 More Replies
YuriS
by > New Contributor II
  • 29 Views
  • 1 replies
  • 0 kudos

StreamingQueryListener metrics strange behaviour (inputRowsPerSecond metric is set to 0)

After implementing StreamingQueryListener to enable integration with our monitoring solution we have noticed some strange metrics for our DeltaSource streams (based on https://learn.microsoft.com/en-us/azure/databricks/structured-streaming/stream-mon...

YuriS_0-1769419735190.png YuriS_1-1769419836870.png
  • 29 Views
  • 1 replies
  • 0 kudos
Latest Reply
hasnat_unifeye
New Contributor III
  • 0 kudos

Firstly -  let’s talk about batch vs trigger.A trigger is the scheduling event that tells Spark when to check for new data (eg processingTime, availableNow, once). A batch (micro-batch) is the actual unit of work that processes data, reads input, and...

  • 0 kudos
hobrob_ex
by > New Contributor II
  • 113 Views
  • 2 replies
  • 0 kudos

Anchor links in notebook markdown

Does anyone know if there is a way to get anchor links working in Databricks notebooks so you can jump to sections in the same book without a full page refresh? i.e. something that works like the following html:<a href="#jump_to_target">Jump</a>...<p...

  • 113 Views
  • 2 replies
  • 0 kudos
Latest Reply
iyashk-DB
Databricks Employee
  • 0 kudos

@hobrob_ex , yes, this is possible, but not like the HTML way; instead, you will have to use the markdown rendering formats. Add #Heading 1, #Heading 2.. so on in the (+Text) button of the notebook. Once these headings/ sections that you want are con...

  • 0 kudos
1 More Replies
discuss_darende
by > Visitor
  • 45 Views
  • 2 replies
  • 1 kudos

Resolved! How can I get workspace groups and their users via a table — and also from a Databricks App?

I’m trying to get a full list of Databricks workspace groups and their user memberships. I want to do this in two ways:As a queryable table or view (e.g., for audits, security reviews, app integration)From within a Databricks App (Streamlit-style), u...

  • 45 Views
  • 2 replies
  • 1 kudos
Latest Reply
Raman_Unifeye
Contributor III
  • 1 kudos

@discuss_darende - you could use below code in the notebook.Pls adjust it based on your need.from databricks.sdk import AccountClient, WorkspaceClient # If env vars are set, this picks them up automatically a = WorkspaceClient() # List identities u...

  • 1 kudos
1 More Replies
francisix
by > New Contributor III
  • 7171 Views
  • 6 replies
  • 9 kudos

Resolved! I haven't received badge for completion

Hi,Today I completed the test for Lakehouse fundamentals by scored 85%, still I haven't received the badge through my email francis@intellectyx.comKindly let me know please !-Francis

  • 7171 Views
  • 6 replies
  • 9 kudos
Latest Reply
sureshrocks1984
New Contributor II
  • 9 kudos

HI  I completed the test for Databricks Certified Data Engineer Associate on 17 December 2024.  still I haven't received the badge through my email sureshrocks.1984@hotmail.comKindly let me know please !SURESHK 

  • 9 kudos
5 More Replies
Brahmareddy
by > Esteemed Contributor
  • 140 Views
  • 2 replies
  • 5 kudos

Data + AI Is Not the Future at Databricks. It’s the Present.

One thing becomes very clear when you spend time in the Databricks community: AI is no longer an experiment. It is already part of how real teams build, ship, and operate data systems at scale.For a long time, many organizations treated data engineer...

  • 140 Views
  • 2 replies
  • 5 kudos
Latest Reply
Brahmareddy
Esteemed Contributor
  • 5 kudos

Thanks @Louis_Frolio for your kind words. Happy to contribute here.

  • 5 kudos
1 More Replies
ADBricksExplore
by > New Contributor
  • 79 Views
  • 1 replies
  • 1 kudos

child Subqueries/sub-statements history metrics, from a parent [CALL...] statement in QueryHistory

Hi,I cannot find so far a way to get programmatically (SQL/Python) the Subqueries(/Sub-statements) executions history records, shown in ADBricks UI Query History/Profile, that were executed during a TaskRun of Job, as shown in [red boxes] on the atta...

image.png
  • 79 Views
  • 1 replies
  • 1 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 1 kudos

Greetings @ADBricksExplore ,  Short answer: there isn’t a supported public API that returns the “Substatements / Subqueries” panel you see in the Query History or Profile UI. The GraphQL endpoints the UI relies on are internal and not stable or suppo...

  • 1 kudos
Danish11052000
by > New Contributor III
  • 164 Views
  • 5 replies
  • 9 kudos

How to get read/write bytes per table using Databricks system tables?

I’m working on a data usage use case and want to understand the right way to get read bytes and written bytes per table in Databricks, especially for Unity Catalog tables.What I wantFor each table, something like:DateTable name (catalog.schema.table)...

  • 164 Views
  • 5 replies
  • 9 kudos
Latest Reply
pradeep_singh
New Contributor II
  • 9 kudos

system.access.audit focuses on governance and admin/security events. It doesn’t capture per-table I/O metrics such as read_bytes or written_bytes.Use system.query.history for per-statement I/O metrics (read_bytes, written_bytes, read_rows, written_ro...

  • 9 kudos
4 More Replies