Databricks Platform Discussions
Dive into comprehensive discussions covering various aspects of the Databricks platform. Join the co...
16471 Posts
Activity in Discussions
- 1676 Views
- 5 replies
- 4 kudos
Resolved! How to retrieve Genie's narrative text summary (final_summary) via API?
Dear Databricks Community,I am trying to programmatically access the narrative text summary that Databricks Genie displays in the UI, but I cannot find it in the public API responses.What I see in the UI: When I ask Genie a question, the UI displays ...
- 1676 Views
- 5 replies
- 4 kudos
Latest Reply
- 4 kudos
Hey, thank you very much! Confirm the text summary is in the response now.
- 4 kudos
- 77 Views
- 4 replies
- 1 kudos
Unable to see a folder in DBFS
Hi Team,We have a few scripts stored in a folder on a DBFS path. Recently, we've noticed that when we navigate to this path manually, the folder appears to be empty, and we are unable to see the scripts.However, the jobs that reference and access the...
- 77 Views
- 4 replies
- 1 kudos
Latest Reply
- 1 kudos
Hi @Shivani_Komma99, Thanks for flagging this. Based on the behaviour you described, this appears to be consistent with a DBFS browser UI issue/regression rather than a problem with the underlying files, especially since the files are still accessibl...
- 1 kudos
- 296 Views
- 8 replies
- 3 kudos
Resolved! Dashboard sharing settings
When attempting to change sharing settings of published dashboards from Only people with access can view to anyone in my account can view or vice versa we get a "Failed to change permissions on dashboard..." error. This happens for all dashboards, bo...
- 296 Views
- 8 replies
- 3 kudos
Latest Reply
- 3 kudos
Wow @Ashwin_DSA that's a quick way to escalate and resolve!
- 3 kudos
- 4203 Views
- 3 replies
- 1 kudos
Resolved! Databricks Connect: Enabling Arrow on Serverless Compute
I recently upgraded my Databricks Connect version to 15.4 and got set up for Serverless, but ran into the following error when I ran the standard code to enable Arrow on Pyspark: >>> spark.conf.set(key='spark.sql.execution.arrow.pyspark.enabled', val...
- 4203 Views
- 3 replies
- 1 kudos
Latest Reply
- 1 kudos
When might this limitation be lifted? Is it due to security concerns that serverless doesn't support this?
- 1 kudos
- 40 Views
- 0 replies
- 0 kudos
Urgent: Installing Lakebridge on Databricks
Hi,I want to install Databricks Lakebridge on my Databricks environment and use the analyze and transpile commands through a python script.The usecase is that we need to create an automated pipeline that will migrate the existing scripts from snowfla...
- 40 Views
- 0 replies
- 0 kudos
- 57 Views
- 0 replies
- 0 kudos
No Egress for Serverless Workspace
Hello,I have a problemimport socket print(socket.gethostbyname("google.com"))[Trace ID: 00-9e5667936e24418dae4c39a9ed003a22-a43a5f2ca9b14451-00] --------------------------------------------------------------------------- gaierror ...
- 57 Views
- 0 replies
- 0 kudos
- 83 Views
- 0 replies
- 0 kudos
Five Unity Catalog ABAC Updates Worth Paying Attention To
ABAC in Unity Catalog already gave teams a better way to apply row filters and column masks at scale. Instead of rebuilding the same access logic table by table, teams can classify data with governed tags and let policies apply automatically across a...
- 83 Views
- 0 replies
- 0 kudos
- 203 Views
- 0 replies
- 1 kudos
How would you design a Spark pipeline to process billions of records efficiently?
Interview Question:Many people start with the row count.I would start with the architecture.Billions of records are not new in enterprise data engineering. The real challenge is designing a pipeline that runs predictably, efficiently, and within SLA....
- 203 Views
- 0 replies
- 1 kudos
- 128 Views
- 0 replies
- 0 kudos
MCP on Databricks
Here’s What Nobody Tells You.A hands-on look at Genie MCP and DBSQL MCP — what works, what doesn’t, and when to skip MCP entirely.Let me start with something most MCP content skips. MCP servers don’t just expose tools. They expose three things — tool...
- 128 Views
- 0 replies
- 0 kudos
- 6889 Views
- 3 replies
- 2 kudos
Deprecation of DBFS?
I am using Azure Databricks DBFS. I have stored some python scripts, notebooks in folders in DBFS root and have mounted ADLS Gen2 to DBFS. Currently I do not have Unity Catalog enabled and this was setup 3-4 years ago. I came across this article http...
- 6889 Views
- 3 replies
- 2 kudos
Latest Reply
- 2 kudos
In current state, when DBFS is disabled in a workspace, both the root filestore and mount paths become inaccessible.What is deprecated are:DBFS root storage the workspace‑provisioned filestore - /dbfs/FileStore/, /dbfs/root/ etc.DBFS mounts (/dbfs/mn...
- 2 kudos
- 485 Views
- 7 replies
- 5 kudos
Resolved! Data Dictionary and Unity Catalog: Best Practices in a Medallion Architecture
Hi everyone,I'm currently working on implementing a data dictionary, and I'm struggling to understand what the best approach is when using a Medallion architecture in Databricks.My company is migrating its data to Databricks so that users can consume...
- 485 Views
- 7 replies
- 5 kudos
Latest Reply
- 5 kudos
Hi, This is a challenging area to get right, especially with so many tables and schemas. Unity Catalog is definitely a good bet for storing this metadata. There are a few approaches that you could take to the data definitions part, and it depends on ...
- 5 kudos
- 105 Views
- 1 replies
- 0 kudos
Code on cluster runs idefinitely
Hello.Ive created a custom cluster - m4.large. When i try to execute some code in this cluster the behaviour is as follows:- Cluster starts, have running status- I run code, for example print("Hello")- Code runs indefinitely- I click interrupt, it st...
- 105 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
The fact that print("Hello") eventually works but SELECT 1 never completes suggests the cluster may be running but not fully initialized for Spark workloads.A few things I’d check first:Cluster Event Log for any provisioning or startup errors.Spark U...
- 0 kudos
- 458 Views
- 4 replies
- 1 kudos
AI Is Becoming a Data Problem Before It Becomes a Model Problem
The conversation around AI often starts with models. In reality, long-term value comes from how well data pipelines, governance, and workflows work together.The teams moving fastest seem to be simplifying their stack instead of expanding it.What tren...
- 458 Views
- 4 replies
- 1 kudos
Latest Reply
- 1 kudos
Coming from a statistics background, this makes a lot of sense to me. Before any model, we always spend most of the time just cleaning and validating data. the actual modeling is usually the smaller part.I think that's why the medallion architecture ...
- 1 kudos
- 121 Views
- 1 replies
- 0 kudos
DAB best practices suggestion
We're currently setting up Databricks Asset Bundles (DAB) with a CI/CD pipeline using Azure DevOps.Our planned development workflow is as follows:Main branch → Developer creates a feature branch → Implement changes → Create a Pull Request → Senior de...
- 121 Views
- 1 replies
- 0 kudos
Latest Reply
- 0 kudos
You can create Databricks Asset Bundles that are decoupled by domain, managed via multi target declarations within configuration and also driven by immutable, versioned artifacts stored securely within Unity Catalog Volumes. You can rely on explicit ...
- 0 kudos
- 204 Views
- 3 replies
- 0 kudos
Bundle deployment overwrites artifacts while jobs are running - best practices?
Hi everyone,I'm using #Declarative Automation Bundles (DAB) to deploy data pipelines, and I've run into an issue with concurrent job runs and deploymentWhat happened:I started a job that depends on a wheel file built by the bundle (timestamped artifa...
- 204 Views
- 3 replies
- 0 kudos
Latest Reply
- 0 kudos
Do you have source_linked_deployment set as false? That's probably causing it.
- 0 kudos
| User | Count |
|---|---|
| 1899 | |
| 1028 | |
| 930 | |
| 480 | |
| 362 |
