cancel
Showing results for 
Search instead for 
Did you mean: 
Community Platform Discussions
Connect with fellow community members to discuss general topics related to the Databricks platform, industry trends, and best practices. Share experiences, ask questions, and foster collaboration within the community.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

VMeghraj
by New Contributor II
  • 1731 Views
  • 1 replies
  • 0 kudos

REST API

Creating an application to capture cluster metrics and sending HTTP REST request to the Spark History Server's API endpoint to retrieve a list of applications. This request doesn't generate logs in the Spark History Server's log files. The Spark Hist...

  • 1731 Views
  • 1 replies
  • 0 kudos
kll
by New Contributor III
  • 2945 Views
  • 1 replies
  • 0 kudos

pass a tuple as parameter to sql query

at_lst = ['131','132','133'] at_tup = (*at_lst,) print(at_tup) ('131','132','133')<div> <div><span>In my sql query, i am trying to pass this on a parameter, however, it doesn't work. <div> <div><div><div><span>%sql<br /><div><span>select * from ma...

  • 2945 Views
  • 1 replies
  • 0 kudos
Latest Reply
kll
New Contributor III
  • 0 kudos

@Retired_mod  I am writing sql using the magic command in the cell block, `%%sql`. Is there a way to pass a parameter in the query without using the `execute` method of the cursor object? Can you please share an example? 

  • 0 kudos
esi
by New Contributor
  • 2517 Views
  • 0 replies
  • 0 kudos

Ingesting PowerBI Tables to databricks

Hi Community,I am looking for a way to access the Power BI tables from databricks and import them as a spark dataframe into my databricks notebook.As far as I have seen, there is a Power BI connector to load data from databricks into Power BI but not...

  • 2517 Views
  • 0 replies
  • 0 kudos
lpf
by New Contributor
  • 1916 Views
  • 0 replies
  • 0 kudos

Changing StreamingWrite API in DBR 13.1 may lead to incompatibility with Spark 3.4

I'm using StarRocks Connector[2] to ingest data to StarRocks on DataBricks 13.1 (powered by Spark 3.4.0). The connector could run on community Spark 3.4, but fail on the DBR. The reason is (the full stack trace is attached)java.lang.IncompatibleClass...

  • 1916 Views
  • 0 replies
  • 0 kudos
lin
by New Contributor
  • 824 Views
  • 0 replies
  • 0 kudos

Facing UNKNOWN_FIELD_EXCEPTION.NEW_FIELDS_IN_FILE

[UNKNOWN_FIELD_EXCEPTION.NEW_FIELDS_IN_FILE] Encountered unknown fields during parsing: [<field_name>], which can be fixed by an automatic retry: trueI am using Azure Databricks, and write with python code. Want to catch the error and raise. Tried wi...

  • 824 Views
  • 0 replies
  • 0 kudos
od
by New Contributor
  • 799 Views
  • 1 replies
  • 0 kudos

How do I regulate notebook cache

I am experiencing error of over caching in databricks notebook. If i display different dfs one of the dfs get cache which after the result of others afterwards. Please how can I avoid the cache memory while using the notebook?

  • 799 Views
  • 1 replies
  • 0 kudos
Latest Reply
-werners-
Esteemed Contributor III
  • 0 kudos

I don't exactly understand what your issue is. Can you elaborate more?

  • 0 kudos
mderela
by New Contributor II
  • 4306 Views
  • 3 replies
  • 1 kudos

inegstion time clustering

Hello, in rerence to https://www.databricks.com/blog/2022/11/18/introducing-ingestion-time-clustering-dbr-112.htmlI have a silly question how to use it. So let's assume that I have a few TB of not partitioned data. So, if I would like to query on dat...

  • 4306 Views
  • 3 replies
  • 1 kudos
Simon_T
by New Contributor III
  • 5221 Views
  • 0 replies
  • 0 kudos

Databricks Terraform Cluster Issue.

Error: default auth: cannot configure default credentials. Config: token=***. Env: DATABRICKS_TOKENon cluster.tf line 27, in data “databricks_spark_version” “latest_lts”:27: data “databricks_spark_version” “latest_lts” {

  • 5221 Views
  • 0 replies
  • 0 kudos
kurtrm
by New Contributor III
  • 5442 Views
  • 2 replies
  • 3 kudos

How to send alert when cluster is running for too long

Hello,Our team recently experienced an issue where a teammate started a new workflow job then went on vacation. This job ended up running continuously without failing for 4.5 days. The usage of the cluster did not seem out of place during the workday...

  • 5442 Views
  • 2 replies
  • 3 kudos
Latest Reply
kurtrm
New Contributor III
  • 3 kudos

@Retired_mod,I ended up creating a job leveraging the Databricks Python SDK to check cluster and active job run times. The script will raise an error and notify the team if the cluster hasn't terminated or restarted in the past 24 hours or if a job h...

  • 3 kudos
1 More Replies
91
by New Contributor
  • 1033 Views
  • 0 replies
  • 0 kudos

Unable to connect to databricks compute server on making first request to the server

Hello all, I am facing an issue while making the first request to the Databricks compute server It will be taking so much time, and in response 504 Gateway timeout error is throwing Could anyone please suggest what is the best possibility we can do?

  • 1033 Views
  • 0 replies
  • 0 kudos

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group
Top Kudoed Authors