Databricks Community

DB1To3

Sorry for the basic question. If I am using an all-purpose clusters in Databricks, it seems to combine all my runs together and execute them within the same Apache Spark application. There is only one application shown in the Spark UI. Unrelated ...

DB1To3 · 03-17-2026

Can someone confirm if this is the right message board for discussing the opensource Apache core of "Spark Connect". (aka databricks connect)We are hosting workloads on Azure Databricks, but would like to ensure that these workloads are following the...

DB1To3

We are hoping to use "spark connect" in the next phase of work and I've heard it isn't trivial to use that feature with "jobs clusters". I will explore both approaches, but I'm guessing that databricks had invested more energy into making that feat...

DB1To3

I will see if there is a way to pass the job name. So far it hasn't shown up in my filter, but it might be an optional parameter that I'm not sending. Thjere is nothing in the filter that corresponds to my runs.I think I'm sending the "run name", b...

DB1To3

I understand about using the API or CLI. However it is very confusing that the list of "jobs" doesn't allow you to filter on a particular compute cluster.Nor does the "all-purpose cluster" allow you to see the list of jobs running on it. In neither...

DB1To3 · 03-22-2026

>> there is no native R UDF pathway over the wire. sparklyr works around this using rpy2, a Python library that embeds and executes R codeThis is interesting. I would not think of python as the best runtime for bridging. I'm wondering if this invol...

DB1To3 · 03-18-2026

@Louis_Frolio Thanks a lot for these details. I am always a little anxious about the roadmap for cloud-hosted services like Spark Connect. I have seen certain Spark-derived technologies get rug-pulled in the past - especially stuff that is built ou...

Databricks Community

User Stats

User Activity

Distinguishing Runs Related to an All-Purpose Cluster

Apache "Spark Connect"

Re: Distinguishing Runs Related to an All-Purpose Cluster

Re: Distinguishing Runs Related to an All-Purpose Cluster

Re: Distinguishing Runs Related to an All-Purpose Cluster

Re: Apache "Spark Connect"

Re: Apache "Spark Connect"