cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 
Help
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Turn on suggestions
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
SB93
by • New Contributor II
  • 225 Views
  • 2 replies
  • 0 kudos

Delta Live Table Pipeline

I have a pipeline that has given me no problems up until today with the following error message:com.databricks.pipelines.common.errors.deployment.DeploymentException: Failed to launch pipeline cluster 0307-134831-tgq587us: Attempt to launch cluster w...

Community Platform Discussions
Reply
  • 225 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
LRALVA
New Contributor III
  • 0 kudos

@SB93 The error message you are seeing indicates that the cluster failed to launch because the Spark driver was unresponsive, with possible causes being library conflicts, incorrect metastore configuration, or other configuration issues. Given that t...

  • 0 kudos
1 More Replies
n1399
by • New Contributor II
  • 118 Views
  • 1 replies
  • 0 kudos

On Demand Pool Configuration & Policy definition

I'm using Job cluster and created compute policies for library management and now I'm trying to use pools in databricks. I'm getting error like this : Cluster validation error: Validation failed for azure_attributes.spot_bid_max_price from pool, the ...

Community Platform Discussions
Reply
  • 118 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
LRALVA
New Contributor III
  • 0 kudos

@n1399 The error "Validation failed for azure_attributes.spot_bid_max_price from pool, the value must be present" suggests that the spot bid max price is required, but it’s either missing or not correctly inherited from the compute policy when using ...

  • 0 kudos
Rachana2
by • New Contributor II
  • 195 Views
  • 3 replies
  • 0 kudos

Databricks lineage

Hello,I am trying to get the table lineage i.e upstreams and downstreams of all tables in unity catalog into my local database using API calls. I need my db to be up to date, if the lineage is updated in one of the in databricks, i have to update sam...

Community Platform Discussions
Reply
  • 195 Views
  • 3 replies
  • 0 kudos
View Replies
Latest Reply
SantoshJoshi
New Contributor II
  • 0 kudos

Hi @Rachana2,As @Alberto_Umana has mentioned I'd check table_lineage / column_lineage tables, as maintaining a lineage through a bespoke pipeline/tooling may not be a right approach.Can you please explain your use case which explains why you don't wa...

  • 0 kudos
2 More Replies
joseroca99
by • New Contributor II
  • 1435 Views
  • 6 replies
  • 0 kudos

Resolved! File found with %fs ls but not with spark.read

Code: wikipediaDF = (spark.read  .option("HEADER", True)  .option("inferSchema", True)  .csv("/databricks-datasets/wikipedia-datasets/data-001/pageviews/raw/pageviews_by_second.tsv"))display(bostonDF) Error: Failed to store the result. Try rerunning ...

Community Platform Discussions
Reply
  • 1435 Views
  • 6 replies
  • 0 kudos
View Replies
Latest Reply
xx123
New Contributor II
  • 0 kudos

I have the exact same issue. Seems like limiting the the display() method works as a temporary solution, but I wonder if there's any long term one. The idea would be to have the possibility of displaying larger datasets within a notebook. How to achi...

  • 0 kudos
5 More Replies
j_h_robinson
by • New Contributor II
  • 150 Views
  • 1 replies
  • 1 kudos

Resolved! Spreadsheet-Like UI for Databricks

We are currently entering data into Excel and then uploading it into Databricks.  Is there a built-in spreadsheet-like UI within Databricks that can update data directly in Databricks? 

Community Platform Discussions
Reply
  • 150 Views
  • 1 replies
  • 1 kudos
View Replies
Latest Reply
Advika_
Databricks Employee
  • 1 kudos

Hello, @j_h_robinson! Databricks doesn’t have a built-in spreadsheet-like UI for direct data entry or editing. Are you manually uploading the Excel files or using an ODBC driver setup? If you’re doing it manually, you might find this helpful: Connect...

  • 1 kudos
h_h_ak
by • Contributor
  • 1168 Views
  • 5 replies
  • 2 kudos

Resolved! Understanding Autoscaling in Databricks: Under What Conditions Does Spark Add a New Worker Node?

I’m currently working with Databricks autoscaling configurations and trying to better understand how Spark decides when to spin up additional worker nodes. My cluster has a minimum of one worker and can scale up to five. I know that tasks are assigne...

Community Platform Discussions
Reply
  • 1168 Views
  • 5 replies
  • 2 kudos
View Replies
Latest Reply
filipniziol
Esteemed Contributor
  • 2 kudos

Hi @h_h_ak ,Short Answer:Autoscaling primarily depends on the number of pending tasks.Workspaces on the Premium plan use optimized autoscaling, while those on the Standard plan use standard autoscaling.Long Answer:Databricks autoscaling responds main...

  • 2 kudos
4 More Replies
Yuki
by • New Contributor III
  • 269 Views
  • 4 replies
  • 4 kudos

Why is the ipynb format recommended?

In this document, https://docs.databricks.com/aws/en/notebooks/notebook-format,Jupyter (.ipynb) format is recommended.> Select File from the workspace menu, select Notebook format, and choose the format you want. You can choose either Jupyter (.ipynb...

Community Platform Discussions
Reply
  • 269 Views
  • 4 replies
  • 4 kudos
View Replies
Latest Reply
Nivethan_Venkat
New Contributor III
  • 4 kudos

Hi @Yuki,One other risk that we foresee / encountered recently is how the notebooks will look in your pull requests of external repos (Azure Devops or GitHub). It will be very hard for a pull request reviewer to understand on the code / notebook read...

  • 4 kudos
3 More Replies
Adil
by • New Contributor
  • 2444 Views
  • 1 replies
  • 0 kudos

Find value in any column in a table

Hi,I'm not sure if this is a possible scenario, but is there, by any chance a way to query all the columns of a table for searching a value? Explanation: I want to search for a specific value in all the columns of a databricks table. I don't know whi...

Community Platform Discussions
Reply
  • 2444 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
sashiDatabricks
New Contributor II
  • 0 kudos

I also have this same requirement now and cant find the solution for this yet. Any help would be good. thanks

  • 0 kudos
lance-gliser
by • New Contributor
  • 531 Views
  • 4 replies
  • 0 kudos

Databricks apps - Volumes and Workspace - FileNotFound issues

I have a Databricks App I need to integrate with volumes using local python os functions. I've setup a simple test:  def __init__(self, config: ObjectStoreConfig): self.config = config # Ensure our required paths are created ...

Community Platform Discussions
Apps
Volumes
Reply
  • 531 Views
  • 4 replies
  • 0 kudos
View Replies
Latest Reply
Datagobes
New Contributor II
  • 0 kudos

If you use the databricks python sdk you can access volume files using built-in app credentials. All you need to do is instantiate the workspace client from the sdk and you can use its methods to operate on volumes.

  • 0 kudos
3 More Replies
Shravanshibu
by • New Contributor III
  • 2460 Views
  • 6 replies
  • 3 kudos

Unable to install a wheel file which is in my volume to a serverless cluster

I am trying to install a wheel file which is in my volume to a serverless cluster, getting the below error@ken@Retired_mod Note: you may need to restart the kernel using %restart_python or dbutils.library.restartPython() to use updated packages. WARN...

Community Platform Discussions
Reply
  • 2460 Views
  • 6 replies
  • 3 kudos
View Replies
Latest Reply
tom-ph
New Contributor II
  • 3 kudos

Same issue here. Any solution?

  • 3 kudos
5 More Replies
pg289
by • New Contributor II
  • 413 Views
  • 0 replies
  • 0 kudos

How to connect to an on-premise implementation of S3 storage (such as Minio) in Databricks Notebooks

I manage a large data lake of Iceberg tables stored on premise in S3 storage from MinIO. I need a Spark cluster to run ETL jobs. I decided to try Databricks as there were no other good options. However, I'm unable to properly access my tables or even...

Community Platform Discussions
Reply
  • 413 Views
  • 0 replies
  • 0 kudos
Sangamswadik
by • New Contributor II
  • 350 Views
  • 4 replies
  • 1 kudos

Resolved! Agents and Inference table errors

Hi, I'm trying to deploy a rag model from GCP databricks. I've added an external gpt4o endpoint and enabled inference table in settings. But when Im trying to deploy agents I'm still getting the inference table not enabled error. (I've registered the...

Sangamswadik_0-1740977753362.png Sangamswadik_1-1740978040054.png
Community Platform Discussions
Reply
  • 350 Views
  • 4 replies
  • 1 kudos
View Replies
Latest Reply
MariuszK
Contributor III
  • 1 kudos

The Model Serving is supported in your region so it can be another problem or limitation.

  • 1 kudos
3 More Replies
Nik_Vanderhoof
by • Contributor
  • 306 Views
  • 2 replies
  • 0 kudos

Resolved! DatabricksWorkflowTaskGroup

Hello,I recently learned about the DatabricksWorkflowTaskGroup operator for Airflow that allows one to run multiple Notebook tasks on a shared job compute cluster from Airflow.Is a similar feature possible to run multiple non-Notebook tasks from Airf...

Community Platform Discussions
Airflow
workflow
Reply
  • 306 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
Nik_Vanderhoof
Contributor
  • 0 kudos

Thank you!

  • 0 kudos
1 More Replies

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Labels
Top Kudoed Authors
View all
Top