cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Tuan
by • New Contributor
  • 427 Views
  • 0 replies
  • 0 kudos

Vector Search

Do you guys think vector search will be cheaper and perform better than Azure AI Search?

Data Engineering
Reply
  • 427 Views
  • 0 replies
  • 0 kudos
Johnc
by • New Contributor
  • 451 Views
  • 0 replies
  • 0 kudos

Connecting to SAP ECC

I'm looking to meet with others who have successfully connected databricks to SAP ECC.

Data Engineering
Reply
  • 451 Views
  • 0 replies
  • 0 kudos
MaOthman2110
by • New Contributor
  • 1553 Views
  • 1 replies
  • 0 kudos

can't login back to databricks community

Hello,I'm experiencing difficulty logging into the Databricks community despite using the correct username and password. Additionally, when attempting to reset my password, I haven't received any email notifications.

Data Engineering
Reply
  • 1553 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
Aurelius
New Contributor II
  • 0 kudos

I am facing the same issue.

  • 0 kudos
Shahe
by • New Contributor
  • 3672 Views
  • 2 replies
  • 0 kudos

Azure Databricks Metrics to Prometheus?

What is the best method to expose Azure Databricks metrics to Prometheus specifically? And is it possible to get the underlying Spark metrics also? All I can see clearly defined in the documentation is the serving endpoint metrics:https://learn.micro...

Data Engineering
Reply
  • 3672 Views
  • 2 replies
  • 0 kudos
View Replies
Latest Reply
DanielB
New Contributor II
  • 0 kudos

HelloI don't have databricks running as pod in an aks cluster.. It's working on azure as saas.. What should I do the export the metrics to prometheus?

  • 0 kudos
1 More Replies
david1144
by • New Contributor III
  • 614 Views
  • 1 replies
  • 0 kudos

Streaming data with kafka

What is the Best way for implement streaming data flow dron kafka to databricks (delta tables)

Data Engineering
Reply
  • 614 Views
  • 1 replies
  • 0 kudos
View Replies
Latest Reply
GCosta
New Contributor II
  • 0 kudos

Structured streaming:https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html

  • 0 kudos
GCosta
by • New Contributor II
  • 871 Views
  • 0 replies
  • 0 kudos

How to write data to Confluent Kafka with SchemaRegistry format on sparkstructured?

Hi There!I am to trying write a batch data to kafka topic with schema registry in databricks using pyspark, i serialize the data with pyspark to_avro function and write it to the topic, but the consumers can’t read the schema id. If they do not separ...

Data Engineering
Reply
  • 871 Views
  • 0 replies
  • 0 kudos
Mlaricobar94
by • New Contributor
  • 599 Views
  • 0 replies
  • 0 kudos

Collect Spark UI statistics to analyze the performance for several Spark Applications

To identify the reasons for a data process poor performance, we need to navigate and analyze the metrics in the Spark UI manually... However, replicating those steps for a giant group of spark applications would be very expensive in times...Given thi...

Data Engineering
Reply
  • 599 Views
  • 0 replies
  • 0 kudos
Serena
by • New Contributor
  • 829 Views
  • 0 replies
  • 0 kudos

Data architecture

Check out our platform architecture italgas-from-gas-pipelines-to-data-pipelines-fueling-our-reporting-with-the-latest-innovations-7f00e20ba115?source=social.linkedin 

Data Engineering
Reply
  • 829 Views
  • 0 replies
  • 0 kudos
Gaurav19
by • New Contributor III
  • 5511 Views
  • 3 replies
  • 1 kudos

Resolved! Databricks API - list job runs doesn't have 'task run id'

Hi all,I am calling get job run list API to get all task ids and refer them in dbt-artifacts view created by dbt job run. The question is I can see 'task run id' on screen but it doesn't come back in api response? Is there a way to get it? I checked ...

Gaurav19_0-1718197141799.png
Data Engineering
Reply
  • 5511 Views
  • 3 replies
  • 1 kudos
View Replies
Latest Reply
Gaurav19
New Contributor III
  • 1 kudos

Never mind, I have found task_run_id present in getrun api https://docs.databricks.com/api/azure/workspace/jobs/getrunI overlooked at first instance as it is buried under nested json structuretasks[] > run_id.This clarifies and solves my problem!

  • 1 kudos
2 More Replies

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now
Labels