cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Community Articles
Dive into a collaborative space where members like YOU can exchange knowledge, tips, and best practices. Join the conversation today and unlock a wealth of collective wisdom to enhance your experience and drive success.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
Kirankumarbs
by • Valued Contributor III
  • 2802 Views
  • 0 replies
  • 1 kudos

How to actually get job_id and run_id in a Databricks Python wheel task (Avoid Hallucinations)

We needed job_id and run_id in a custom metrics Delta table so we could join to `system.lakeflow.job_run_timeline`. Tried four approaches before finding the one that works on serverless compute.What doesn't workspark.conf.get("spark.databricks.job.id...

Community Articles
Reply
  • 2802 Views
  • 0 replies
  • 1 kudos
Kirankumarbs
by • Valued Contributor III
  • 815 Views
  • 1 replies
  • 2 kudos

Multi-Task on a Shared Cluster — Why That's Also Not Enough

Part 2 of 3 — Databricks Streaming ArchitectureThe instinct after Part 1 was obvious.If running eight queries in one task means one failure can hide while others keep running — split them into multiple tasks. Separate concerns. Give each component it...

Community Articles
Reply
  • 815 Views
  • 1 replies
  • 2 kudos
View Replies
Latest Reply
Kirankumarbs
Valued Contributor III
  • 2 kudos

Part 1: Streaming Failure Models: Why "It Didn't Crash" Is the Worst OutcomePart 3: One Cluster per Task — Proven, Ready, and Waiting

  • 2 kudos
Labels