cancel
Showing results for 
Search instead for 
Did you mean: 
Databricks Platform Discussions
Dive into comprehensive discussions covering various aspects of the Databricks platform. Join the conversation to deepen your understanding and maximize your usage of the Databricks platform.
cancel
Showing results for 
Search instead for 
Did you mean: 

Browse the Community

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies with...

12175 Posts

Data Governance

Join discussions on data governance practices, compliance, and security within the Databricks Commun...

527 Posts

Generative AI

Explore discussions on generative artificial intelligence techniques and applications within the Dat...

374 Posts

Machine Learning

Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithm...

1019 Posts

Warehousing & Analytics

Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Communi...

682 Posts

Activity in Databricks Platform Discussions

Ashwin_DSA
by Databricks Employee
  • 4 Views
  • 0 replies
  • 0 kudos

Is Address Line 4 the place where data goes to die?

I’ve spent the last few years jumping between insurance, healthcare, and retail, and I’ve come to a very painful conclusion that we should never have let humans type their own addresses into a text box.  For a pet project, I’m currently looking at a ...

  • 4 Views
  • 0 replies
  • 0 kudos
vziog
by > New Contributor III
  • 493 Views
  • 4 replies
  • 0 kudos

Databricks Default Catalog with House Icon

Hi everyone,I have a question regarding workspace catalogs in Databricks with Unity Catalog.In our setup, when a new workspace is created and automatically assigned to a Unity Catalog metastore (with automatic workspace catalog creation enabled), a c...

vziog_0-1772791652417.png
  • 493 Views
  • 4 replies
  • 0 kudos
Latest Reply
SteveOstrowski
Databricks Employee
  • 0 kudos

Hi @vziog, I can see the earlier suggestion about using the kebab menu Rename didn't fully solve the problem. There are actually a few distinct concepts at play here, and understanding the difference is the key. WHAT IS THE WORKSPACE CATALOG (HOUSE I...

  • 0 kudos
3 More Replies
397973
by > New Contributor III
  • 30 Views
  • 1 replies
  • 0 kudos

Is it unusual that I need to start a compute cluster to sync with Git?

I would guess unusual but want to hear from others before I nag my managers about it. In Databricks (I access in web browser) we have a compute cluster specifically for Git; you need to start it to push code or even to change branches. This is separa...

  • 30 Views
  • 1 replies
  • 0 kudos
Latest Reply
MoJaMa
Databricks Employee
  • 0 kudos

It means you are on the old classic Git Proxy that helped establish connectivity from the Databricks Control Plane to your on-prem Git Server. If your Git Server was cloud-based you would not need the proxy cluster. That being said, the new way is th...

  • 0 kudos
DineshOjha
by > New Contributor II
  • 54 Views
  • 1 replies
  • 0 kudos

Service Principal access notebooks created under /Workspace/Users

What permissions does a Service Principal need to run Databricks jobs that reference notebooks created by a user and stored in Git?Hi everyone,We are exploring the notebooks‑first development approach with Databricks Bundles, and we’ve run into a wor...

  • 54 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ashwin_DSA
Databricks Employee
  • 0 kudos

Hi @DineshOjha, This is a good question, and researching this helped me learn some best practices along the way. What you’re seeing is actually expected behaviour. Service principals aren’t meant to execute notebooks directly from users’ personal wor...

  • 0 kudos
Phani1
by > Databricks MVP
  • 68 Views
  • 1 replies
  • 0 kudos

Databricks Cost Estimation Template

Hi Databricks Team, Is there a standard Databricks cost estimation template(xl), sizing calculator, or TCO tool that allows us to provide the following inputs and derive an approximate monthly and annual platform cost:Source systems and their types (...

  • 68 Views
  • 1 replies
  • 0 kudos
Latest Reply
emma_s
Databricks Employee
  • 0 kudos

Hi, There isn't anything publicly available that I'm aware of. For this kind of complex migration I'd recommend working with your account team. As somebody who does Databricks sizing a lot, it's a nuanced art which I suspect is why we don't have any ...

  • 0 kudos
SunilN
by > New Contributor
  • 24 Views
  • 1 replies
  • 0 kudos

New policy for mask column fails - Compilation error with message 'Unknown tag policy key'

I have created custom tags on a column and plan to mask columns with tags via policy.  I am facing 2 issues 1. Cant see the custom tag under - Mask column if it has specific tag. 2. If I type my custom tag get error when creating policy  Policy creat...

  • 24 Views
  • 1 replies
  • 0 kudos
Latest Reply
emma_s
Databricks Employee
  • 0 kudos

Hi, to use a tag in a tag policy it needs to be a governed tag rather than just a general tag. IF you just create it using Set tags UC sees it as a informational tag rather than a governed tag. If you use a CREATE tag statement to create it then you'...

  • 0 kudos
PradeepPrabha
by > New Contributor III
  • 2071 Views
  • 6 replies
  • 0 kudos

Resolved! Any documentation mentioning connectivity from Azure SQL database connectivity to Azure Databricks

Any documentation available to connect from the Azure SQL database to Azure Databricks SQL workspace. We created a SQL warehouse personal access token for a user in a different team who can connect from his on-prem SQL DB to Databricks using the conn...

  • 2071 Views
  • 6 replies
  • 0 kudos
Latest Reply
PradeepPrabha
New Contributor III
  • 0 kudos

Thank you for the detailed answer

  • 0 kudos
5 More Replies
PradeepPrabha
by > New Contributor III
  • 207 Views
  • 2 replies
  • 1 kudos

Resolved! Any recommended way for a different app to start their dependent job based on Databricks job?

How can we configure a job in a different Azure application to be triggered after the completion of an Azure Databricks job? Once the Databricks job is successful, the job in the third-party application hosted in Azure should start. I attempted to us...

  • 207 Views
  • 2 replies
  • 1 kudos
Latest Reply
PradeepPrabha
New Contributor III
  • 1 kudos

Thank you.  Thank you for the detailed answer!I have tested the Azure function way and also using an Azure runbook as well. Both works fine.Also tested the option of adding as the final task and a condition to "if all other notebooks" successful, the...

  • 1 kudos
1 More Replies
rtglorenabasul
by > Visitor
  • 22 Views
  • 1 replies
  • 0 kudos

Issue Running Job on Serverless GPU

I have a job that runs a notebook, the notebook uses serverless GPU (A10) and it keeps failing with a "Run failed with error message Cluster 'xxxxxxxxxxx' was terminated. Reason: UNKNOWN (SUCCESS)". The base environment is 'Standard v4' and I have tr...

  • 22 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ashwin_DSA
Databricks Employee
  • 0 kudos

Hi @rtglorenabasul, Thanks for sharing the details. The behaviour you’re seeing is consistent with an issue in how the job is bringing up Serverless GPU compute, rather than with the notebook code itself. Having done some checks, that error usually m...

  • 0 kudos
kcheng
by > Visitor
  • 36 Views
  • 1 replies
  • 1 kudos

Resolved! Massive Duplicate Alerts Auto‑Created by Git Folders After Recent Databricks Update

Hello Databricks Support Team,I’m experiencing a severe issue in my Databricks workspace related to the new Git‑Integrated Alerts behavior. Overnight, my workspace went from 67 alerts to nearly 1,000 alerts, all of which appear to have been auto‑gene...

  • 36 Views
  • 1 replies
  • 1 kudos
Latest Reply
Ashwin_DSA
Databricks Employee
  • 1 kudos

Hi @kcheng, Thanks for sharing the details. This looks like behaviour that will need workspace‑specific investigation by Databricks Support, rather than something the community can reliably diagnose or fix. Because it resulted in a sudden, large volu...

  • 1 kudos
AanchalSoni
by > Contributor
  • 74 Views
  • 4 replies
  • 1 kudos

Resolved! Checkpoint Location Error

 Hi!I'm facing an error related to Checkpoint whenever I try to display a dataframe using auto Loader in Databricks free edition. Please refer the screenshot. To combat this, I have to delete the checkpoint folder and then execute the display or writ...

  • 74 Views
  • 4 replies
  • 1 kudos
Latest Reply
Ashwin_DSA
Databricks Employee
  • 1 kudos

Hi @AanchalSoni, I can’t see the full history of your notebook, so I’m not sure of the exact cause. But the behaviour strongly suggests that an earlier version of the stream used complete mode against the same checkpointLocation, and that configurati...

  • 1 kudos
3 More Replies
souravg
by > Visitor
  • 33 Views
  • 2 replies
  • 1 kudos

Feedback not showing up in Genie from Copilot Studio Genie Agent

Hi,We've created an Agent using Copilot Studio for Genie and integrated with Teams Channel.The feedback there is working and we can see the reactions in the Copilot Studio Analytics.But the feedback is not going to the actual genie space, neither the...

  • 33 Views
  • 2 replies
  • 1 kudos
Latest Reply
Ashwin_DSA
Databricks Employee
  • 1 kudos

Hi @souravg, @Ale_Armillotta is right. At the moment, Genie only records feedback (thumbs up/down, "Fix it", comments) when it’s given directly in the Genie UI. The public Genie Conversation APIs that Copilot Studio/Teams use don’t expose any endpoin...

  • 1 kudos
1 More Replies
vidya_kothavale
by > Contributor
  • 5018 Views
  • 5 replies
  • 3 kudos

Resolved! Issue with MongoDB Spark Connector in Databricks

 I followed the official Databricks documentation("https://docs.databricks.com/en/_extras/notebooks/source/mongodb.html")to integrate MongoDB Atlas with Spark by setting up the MongoDB Spark Connector and configuring the connection string in my Datab...

  • 5018 Views
  • 5 replies
  • 3 kudos
Latest Reply
NaveenBedadala
New Contributor II
  • 3 kudos

try using single-user cluster or non-isolation cluster instead of shared cluster

  • 3 kudos
4 More Replies
AanchalSoni
by > Contributor
  • 57 Views
  • 2 replies
  • 2 kudos

Resolved! NULL rows getting inserted in delta table- Schema mismatch

I'm trying to add _metadata column while reading a json file: from pyspark.sql.functions import colfrom pyspark.sql.types import StructType, StructField, LongType, TimestampTypedf_accounts_read = spark.readStream.format("cloudFiles").\    option("clo...

  • 57 Views
  • 2 replies
  • 2 kudos
Latest Reply
Ashwin_DSA
Databricks Employee
  • 2 kudos

Hi @AanchalSoni, Looking at the first snapshot, it appears the path in all three records points to the checkpoint location. The _metadata column isn’t the root cause here. The issue is that Autoloader is ingesting your checkpoint files as data. Becau...

  • 2 kudos
1 More Replies
IM_01
by > Contributor
  • 113 Views
  • 9 replies
  • 3 kudos

Resolved! Partition cols for a temporary table in Lakefow SDP

Hi,I was going through the documentation on quarantining records. Initially I thought that partitioning is not supported for temporary tables however I came cross the following@DP.table( temporary=True, partition_cols=["is_quarantined"], ) @dp.ex...

  • 113 Views
  • 9 replies
  • 3 kudos
Latest Reply
IM_01
Contributor
  • 3 kudos

Thanks Ashwin

  • 3 kudos
8 More Replies