cancel
Showing results for 
Search instead for 
Did you mean: 
Databricks Platform Discussions
Dive into comprehensive discussions covering various aspects of the Databricks platform. Join the conversation to deepen your understanding and maximize your usage of the Databricks platform.
cancel
Showing results for 
Search instead for 
Did you mean: 

Browse the Community

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies with...

11572 Posts

Data Governance

Join discussions on data governance practices, compliance, and security within the Databricks Commun...

491 Posts

Generative AI

Explore discussions on generative artificial intelligence techniques and applications within the Dat...

282 Posts

Machine Learning

Dive into the world of machine learning on the Databricks platform. Explore discussions on algorithm...

976 Posts

Warehousing & Analytics

Engage in discussions on data warehousing, analytics, and BI solutions within the Databricks Communi...

632 Posts

Activity in Databricks Platform Discussions

JerryAnderson
by > Visitor
  • 5 Views
  • 0 replies
  • 0 kudos

Need to claim Azure Databricks account for workspace created via Resource Provider

Hello,Our Azure Databricks workspace (URL: https://adb-3568788088379780.0.azuredatabricks.net) was deployed by the Azure Databricks Resource Provider. No “Manage Account” option appears in the UI, and no Account Admin is listed. Please link this work...

  • 5 Views
  • 0 replies
  • 0 kudos
PearceR
by > New Contributor III
  • 49 Views
  • 1 replies
  • 0 kudos

Service Principal with Federated Credentials Can’t Access Full Repo in ADO

Good Afternoon,I’m using Databricks with Git integration to Azure DevOps (ADO).Authentication is via Microsoft Entra federated credentials for a service principal (SPN).The SPN has Basic access in ADO, is in the same project groups as my user, and Gi...

  • 49 Views
  • 1 replies
  • 0 kudos
Latest Reply
AbhaySingh
Databricks Employee
  • 0 kudos

The issue stems from a fundamental architectural difference in how Databricks handles Git authentication:     1. Git Credential Gap: While your SPN successfully authenticates to Databricks via Microsoft Entra federated   credentials, it lacks the sec...

  • 0 kudos
RakeshRakesh_De
by > New Contributor III
  • 2287 Views
  • 3 replies
  • 1 kudos

Databricks Free Edition - sql server connector not working-

I am trying to explore New Databricks Free edition but SQL Server connector Ingestion pipeline not able to set up through UI.. Its showing error that --Serverless Compute Must be Enabled for the workspace,But Free Edition only have Serverless Option ...

Data Engineering
FreeEdition
LakeFlow
  • 2287 Views
  • 3 replies
  • 1 kudos
Latest Reply
Saf4Databricks
New Contributor III
  • 1 kudos

Hi @RakeshRakesh_De  The error is misleading. As mentioned in the second row of the table here the gateway runs on classic compute, and the ingestion pipeline runs on serverless compute (mentioned in the third row of the same table linked above). Hop...

  • 1 kudos
2 More Replies
austinoyoung
by > New Contributor III
  • 20 Views
  • 2 replies
  • 1 kudos

Resolved! oracle sequence number

Dear All,I am trying to use jdbc driver to connect to an oracle database and append a new record to a table. The table has a column needs to be populated with a sequence number. I've been trying to use select `<sequence_name>.nextval` to get the sequ...

  • 20 Views
  • 2 replies
  • 1 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 1 kudos

Hey @austinoyoung ,  Short answer: Don’t try to pull the sequence in your Spark insert. Let Oracle assign it. Why this happens (ORA-02287: sequence number not allowed here Spark’s JDBC writer generates parameterized INSERT statements like: INSERT INT...

  • 1 kudos
1 More Replies
refah_1
by > New Contributor
  • 3101 Views
  • 1 replies
  • 0 kudos

von Google Cloud Storage

Hi everyone,I'm new to Databricks and am trying to connect my Google Cloud Storage bucket to my Databricks workspace. I have a 43GB CSV file stored in a GCP bucket that I want to work with. Here’s what I've done so far:Bucket Setup:I created a GCP bu...

  • 3101 Views
  • 1 replies
  • 0 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 0 kudos

Hey @refah_1 ,  Thanks for laying out the steps—you’re very close. Here’s a structured checklist to get GCS working with Unity Catalog and a couple of common gotchas to check.   What’s likely going on The region mismatch isn’t the root cause; docs em...

  • 0 kudos
borft
by > New Contributor
  • 2888 Views
  • 1 replies
  • 0 kudos

Databricks on GCP admin console access

Hi,I'm trying to update the GCP permissions for Databricks as described here: https://docs.databricks.com/gcp/en/admin/cloud-configurations/gcp/gce-updateTo be able to do that, I have to log in to the account console here: https://accounts.gcp.databr...

  • 2888 Views
  • 1 replies
  • 0 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 0 kudos

Greetings @borft ,  It sounds like you’re being redirected into a workspace without the right privileges; let’s get you into the correct Databricks account console for your GCP Marketplace subscription and identify the right login. What login is requ...

  • 0 kudos
Sven_Relijveld
by > New Contributor II
  • 246 Views
  • 3 replies
  • 2 kudos

Resolved! Programmatically activate groups in account

Hi,I am currently trying to use the Accounts SDK to add External groups from Entra ID to functional groups within Databricks. I expect thousands of groups in Entra and I want to add these groups programmatically (for example) to a group in Databricks...

  • 246 Views
  • 3 replies
  • 2 kudos
Latest Reply
Louis_Frolio
Databricks Employee
  • 2 kudos

Hi @Sven_Relijveld — great to hear that your bulk-initial activation workflow is working as expected. Thanks for the update. Regarding the 5K external group limit you’re seeing: That is the current default soft quota for Azure Databricks accounts. It...

  • 2 kudos
2 More Replies
ma10
by > New Contributor
  • 4133 Views
  • 6 replies
  • 1 kudos

Issue with updating email with SCIM Provisioning

Hi all,For our set-up we have configured SCIM provisioning using Entra ID, group assignment on Azure is dealt with by IdentityIQ Sailpoint, and have enabled SSO for Databricks. It has/is working fine apart from one scenario. The original email assign...

  • 4133 Views
  • 6 replies
  • 1 kudos
Latest Reply
nayan_wylde
Esteemed Contributor
  • 1 kudos

The other option is to raise a ticket with Databricks Accounts team. Our Databricks team worked on the backend and the new email was synced.

  • 1 kudos
5 More Replies
bidek56
by > New Contributor III
  • 14 Views
  • 1 replies
  • 0 kudos

Resolved! Stack traces as standard error in job logs

When using DBR 16.4, I am seeing a lot of Stack traces as standard error in jobs, any idea why they are showing up and how to turn then off? Thx"FlagSettingCacheMetricsTimer" id=18 state=WAITING- waiting on <0x2d1573c6> (a java.util.TaskQueue)- locke...

  • 14 Views
  • 1 replies
  • 0 kudos
Latest Reply
bidek56
New Contributor III
  • 0 kudos

spark.databricks.driver.disableJvmThreadDump=trueThis setting will remove the ST. 

  • 0 kudos
crami
by > New Contributor II
  • 12 Views
  • 1 replies
  • 0 kudos

Quota Limit Exhausted Error when Creating declarative pipeline

I am trying to develop a declarative pipeline. As per platform policy, I cannot use serverless, reason, I am using asset bundle to create declarative pipeline. In the bundle, I am trying to specify compute for the pipeline. However, I am constantly f...

crami_1-1761925275134.png crami_0-1761925248664.png crami_2-1761925397717.png
  • 12 Views
  • 1 replies
  • 0 kudos
Latest Reply
Khaja_Zaffer
Contributor III
  • 0 kudos

Hello @crami Good day!!As the error tells. you need to increase the VM size, i know you have enough things in your place but spot fallback + Photon + autoscale triggers the failure.   Go to Azure Portal → Subscriptions → Usage + quotasFilter: Provide...

  • 0 kudos
cleversuresh
by > New Contributor III
  • 4197 Views
  • 1 replies
  • 0 kudos

AutoGluon MLflow integration

I am working on a personalized price package recommendation and implemented an AutoGluon code integrating it with MLflow.The code has been created in a modular fashion to be used by other team members. They just need to pass the data, target column a...

  • 4197 Views
  • 1 replies
  • 0 kudos
Latest Reply
stbjelcevic
Databricks Employee
  • 0 kudos

Hi @cleversuresh  Thanks for sharing the code and the context. Here are the core issues I see and how to fix them so MLflow logging works reliably on Databricks. What’s breaking MLflow logging in your code Your PyFunc wrapper loads the AutoGluon mod...

  • 0 kudos
dkxxx-rc
by > Contributor
  • 3487 Views
  • 1 replies
  • 0 kudos

AutoML master notebook failing

I have recently been able to run AutoML successfully on a certain dataset.  But it has just failed on a second dataset of similar construction, before being able to produce any machine learning training runs or output.  The Experiments page says```Mo...

dkxxxrc_0-1740403690249.png
  • 3487 Views
  • 1 replies
  • 0 kudos
Latest Reply
stbjelcevic
Databricks Employee
  • 0 kudos

Hi @dkxxx-rc , Thanks for the detailed context. This error is almost certainly coming from AutoML’s internal handling of imbalanced data and sampling, not your dataset itself. The internal column _automl_sample_weight_0000 is created by AutoML when i...

  • 0 kudos
saab123
by > New Contributor II
  • 3223 Views
  • 1 replies
  • 0 kudos

Not able to export maps in dashboards

When we export a dashboard with maps, the map background doesn't show up in the pdf. 

  • 3223 Views
  • 1 replies
  • 0 kudos
Latest Reply
mark_ott
Databricks Employee
  • 0 kudos

When exporting a Databricks dashboard with maps to PDF, it is a known issue that the map background sometimes does not appear in the exported PDF file. This problem has been discussed in the Databricks community as of early 2025, and appears to be a ...

  • 0 kudos
SrihariB
by > New Contributor
  • 3661 Views
  • 1 replies
  • 0 kudos

Read from multiple sources in a single stream

Hey all, I am trying to read data from multiple s3 locations using a single stream DLT pipeline and loading data into a single target. Here is the scenario. S3 Locations: Below are my s3 raw locations with change in the directory names at the end. Ba...

  • 3661 Views
  • 1 replies
  • 0 kudos
Latest Reply
mark_ott
Databricks Employee
  • 0 kudos

You are using Databricks Autoloader (cloudFiles) within a Delta Live Tables (DLT) pipeline to ingest streaming Parquet data from multiple S3 directories with a wildcard pattern, and you want to ensure all matching directories’ data is included in a s...

  • 0 kudos
Mauro
by > New Contributor II
  • 3217 Views
  • 1 replies
  • 0 kudos

DLT change in hive metastore destination to unity catalog

A change recently came out in which Databricks necessarily requires using the Unity Catalog as the output of a DLT because previously it was HiveMetaStore. At first I was working using CDC plus expectations which resulted in the "allow_expectations_c...

  • 3217 Views
  • 1 replies
  • 0 kudos
Latest Reply
mark_ott
Databricks Employee
  • 0 kudos

Databricks has recently enforced Unity Catalog as the output target for Delta Live Tables (DLT), replacing the legacy Hive Metastore approach. As a result, the familiar "allow_expectations_col" column, which was automatically added to help track and ...

  • 0 kudos