Databricks Community

Vamsi3757

Hi Everyone,I'm building a Databricks cost optimization agent and would appreciate some guidance on the recommended architecture.My current design consists of the following functions:Retrieve the last N job runs along with cost, DBU usage, runtime, c...

Vamsi3757

Hi Everyone,I'm trying to capture Spark Event Logs for a Databricks cost optimization solution.I configured my job cluster as follows:spark.eventLog.enabled=truespark.eventLog.dir=dbfs:/databricks/spark-events/The cluster starts successfully and I ca...

Vamsi3757 · 05-28-2026

Hello everyone,I have built an ADK-based agent that connects to Databricks and can retrieve various information. However, I’m trying to design a secure way to authenticate each user individually when they interact with the agent.So far, I see two pos...

Vamsi3757

Thanks for the explanation. Just to confirm, if I configure Compute Log Delivery to a Unity Catalog Volume, will the event log/ folder contain the actual Spark Event Logs (jobs, stages, tasks, executor metrics, etc.) that can be parsed and analyzed, ...

Databricks Community

User Stats

User Activity

Best Architecture for Building a Custom Agent in Databricks?

Spark Event Logging to DBFS creates only 0-byte .inprogress file

Best Approach for Per-User Authentication with Databricks in an Agent-Based System

Re: Spark Event Logging to DBFS creates only 0-byte .inprogress file