rss.livelink.threads-in-node

Tuning with Optuna and MlflowSparkStudy

AdamIH123 — Mon, 07 Sep 2026 19:24:05 GMT

I am following the guide for tuning a model with Optuna and MlflowSparkStudy. My compute is configured with autoscaling enabled, with 1–2 Spark workers, each with 8 cores and 32 GB of memory. I set n_jobs=2 and trials=100, in mlflow_study.optimize().

I have a few questions about how MlflowSparkStudy distributes the workload:

How should I think about n_jobs in mlflow_study.optimize() vs. num_threads/n_jobs in LightGBM?
For example, with 1–2 Spark workers and 8 cores per worker, would it make sense to set mlflow_study.optimize(n_jobs=2) and LightGBM n_jobs=7? My dataset is large, so ideally I would like to run only one model-training job per Spark worker and use the remaining cores on that worker for the LightGBM training.
Does MlflowSparkStudy automatically trigger or make use of Spark autoscaling?
If I configure my cluster with a minimum of 1 worker and a maximum of 2 workers, will MlflowSparkStudy cause Spark to scale up to 2 workers as needed when running multiple trials in parallel?
What is the recommended way to make a large DataFrame available to each Spark worker?
Currently, I have a Pandas DataFrame that I pass to the objective function used by mlflow_study.optimize(). Should I broadcast the DataFrame, cache it in Spark, or use another approach to avoid repeatedly transferring the data to each worker for every trial?

Any guidance on the recommended configuration or best practices would be greatly appreciated.

#mlflow #optuna #MlflowSparkStudy

https://learn.microsoft.com/en-us/azure/databricks/machine-learning/automl-hyperparam-tuning/optuna

How to extract table-level execution time and resource allocation within a multi-table Job?

Dolly0503 — Mon, 07 Sep 2026 05:36:11 GMT

Hi everyone,

I am working on calculating accurate compute costs and execution times for individual tables within our Databricks environment, but I am running into an issue with metric granularity.

Currently, we are fetching execution data based on job_id. The problem is that a single job_id often executes multiple tables of varying sizes. Right now, our logic can only derive an average execution time across the entire job, which is highly inaccurate for attributing costs to a specific table.

To get precise, table-level cost metrics, we need to move away from job-level averages. Specifically, I am trying to find a way to extract:

Table-Specific Timestamps: The exact execution start_time and end_time for an individual table running within a larger job.
Granular Resource Allocation: The exact number of compute resources (DBUs, allocated vs. free resources) consumed specifically during the time that individual table is actively running.

Are there specific System Tables (e.g., within system.information_schema or system.access), REST API endpoints, or Spark listener configurations that expose this level of granular, table-specific execution data? and also below am attaching my query.

WITH cost_agg AS (
SELECT job_id, day, SUM(cost_consumed) AS cost_val
FROM costing --Internal table
GROUP BY job_id, day
--goes to costing dashboard table for each job, for each day - adds up all the cost. Result is one row per job per day with total cost
),
base_data AS (
SELECT job_id, task_key,
to_date(period_start_time) as execution_date,
execution_duration_seconds
FROM system.lakeflow.job_task_run_timeline

),
task_avgs AS (
SELECT job_id, task_key, execution_date, avg(execution_duration_seconds) as avg_task_exec_for_day
FROM base_data
GROUP BY job_id, execution_date, task_key
--For each task within each job on each day, averages the execution time across all runs of that task that day.
-- e.g. prd-customers ran 3 times: 100s + 120s + 80s → avg = 100s
),
job_totals as (
select job_id, execution_date, greatest(sum(avg_task_exec_for_day),1) as total_task_seconds_for_day
from task_avgs
group by job_id, execution_date
--Sums all task averages per job per day to get the denominator for weightage.
),
table_costs as ( SELECT
ta.job_id, ta.execution_date, ta.task_key,
SPLIT_PART(ta.task_key, '-', 1) AS ctlg,
SPLIT_PART(ta.task_key, '-', 2) AS db_name,
SPLIT_PART(ta.task_key, '-', 3) AS tbl_name,
round(ta.avg_task_exec_for_day/jt.total_task_seconds_for_day, 5),
round(c.cost_val, 5) as job_cost_usd,
round((ta.avg_task_exec_for_day/jt.total_task_seconds_for_day) * c.cost_val, 5) as table_cost_usd
from task_avgs ta
join job_totals jt on ta.job_id = jt.job_id AND ta.execution_date = jt.execution_date
left join cost_agg c on c.job_id = ta.job_id AND c.day = ta.execution_date
)

select ctlg, db_name, tbl_name,
array_join(collect_set(cast(job_id as string)),',') as jobs_id,
count(distinct job_id) as jobs_count,
count(*) as times_run,
count(distinct execution_date) as days_run,
min(execution_date) as first_run_date,
max(execution_date) as last_run_date,
round(sum(table_cost_usd),5) as total_cost_usd,
ROUND(SUM(table_cost_usd) / NULLIF(COUNT(DISTINCT execution_date), 0), 5) AS avg_daily_cost_usd, -- ← replaced here
round(min(table_cost_usd),5) as min_daily_cost_usd,
round(max(table_cost_usd),5) as max_daily_cost_usd
from table_costs
where execution_date <= '2026-03-04' and ctlg = 'prd'
group by tbl_name, ctlg, db_name
order by total_cost_usd desc

Any guidance, query examples, or best practices would be greatly appreciated.

Thank you!

Understanding Parquet File Storage for Large Datasets

gowri_databrick — Mon, 07 Sep 2026 04:23:05 GMT

Hi everyone,

I’m learning about Parquet files and how they are used in Databricks for storing large datasets.

I’m trying to understand how column-based storage works in a practical situation.

For example, suppose an e-commerce company has 500 million order records containing customer details, product information, order dates, and payment information. If an analyst only needs order_date and order_amount to calculate daily sales, how does storing the data in Parquet help the system process this query efficiently?

I’d like to understand how Parquet storage works in this type of real-world scenario.

Thanks!

What is a Checkpoint in Structured Streaming?

gowri_databrick — Sun, 06 Sep 2026 12:55:47 GMT

Hi everyone,

I’m learning about Structured Streaming in Databricks and came across checkpoints.

I understand that checkpoints are used to keep track of the progress of a streaming query, but I’d like to understand their purpose more clearly.

For example, if a streaming pipeline is processing customer transactions and the pipeline stops unexpectedly, how does the checkpoint help the pipeline continue processing from where it stopped?

What is the main purpose of checkpoints, and why are they important in a real-time data pipeline?

Thanks!

Lakeflow SDP Append Flow

IM_01 — Sun, 06 Sep 2026 09:23:58 GMT

Hi All,
I'm using append_flow to ingest data into the target table. Before returning the dataframe, I compute few column trnasformations, but those values aren't being calculated correctly (or: aren't showing up at all)
so the append flow should not include any column transformations.Does it need to include only reading source table?

Databricks-Native AI Agent for Job Incident Detection, RCA & Safe Remediation

VibinRoy_C — Sat, 05 Sep 2026 11:29:57 GMT

I’m exploring an architecture for a Databricks-native AI Agent for intelligent Spark job incident detection, root cause analysis (RCA), and safe remediation, and I would love your technical feedback.

The Problem

A common operational challenge is the "Silent Long-Runner" or stalled workload—for example, a Spark job that normally finishes in ~1 hour suddenly runs for 8+ hours with zero meaningful progress, wasting DBUs and blocking pipelines.

Currently, investigating this requires engineers to manually correlate disparate data sources:

Spark UI: Job/Stage DAGs, Task skew, stragglers, shuffle memory/disk spill.
Compute State: Driver/Executor availability, OOMs, memory pressure, thread dumps.
Logs & I/O: Driver/executor logs and real-time input/output progression.
Historical Context: Baseline metrics from past successful runs.

Proposed Concept: Execution-Aware Agentic Intelligence

Instead of a simple log-parsing chatbot, the goal is a closed-loop agent framework built on Databricks-native capabilities (System Tables, MLflow, Agent Framework/MCP, Unity Catalog):

Baseline Learning: Dynamically detects anomalies against historical execution patterns instead of static time limits.
Multi-Signal RCA & Evidence Chain: Correlates live Spark UI metrics, logs, and compute telemetry to yield an explainable diagnosis (e.g., "Root Cause: Executor OOM & Disk Spill Skew — 92% Confidence").
Policy-Governed Remediation: Uses an Observe → Recommend → Auto-Remediate model, requiring Human-in-the-Loop (HITL) approval for high-risk actions (e.g., job cancellation, cluster resizing).
Closed Feedback Loop: Captures whether actions resolved the issue to continuously improve accuracy and quantify DBU/cost savings.

Questions for the Community

Telemetry Access: How much granular Spark UI data (stage progress, task skew, shuffle details) can be programmatically accessed in real time via System Tables or APIs?
Native Pattern: What native capabilities would you recommend for correlating these runtime and log signals efficiently?
Remediation Safety: What guardrails or design patterns do you recommend for automated or HITL actions on running jobs?
Prior Art: Has anyone built or tested a similar agentic incident detection workflow inside Databricks?

Would love to hear your thoughts, feedback, or architectural suggestions!

#genAI #dataengineering #AIagent

Databricks vs Snowflake Pyspark Performance

Sam500 — Sat, 05 Sep 2026 10:08:20 GMT

Hi experts, now the competition between cloud providers are fierce and brutal , I come across this post which compares Pyspark performance on databricks and Snowflake

https://l1nk.dev/1y33h0z although the metics tested are not rigorous and exhaustive, but comparative results has been shared. I want to know whether do we have similar comparisons on larger scale with wider set of benchmarks so when these discussions come up before prospective client, we have solid comparison resultset ready to share. Thank you in advance.

What is a Data Skipping in Delta Lake?

gowri_databrick — Sat, 05 Sep 2026 08:02:08 GMT

Hi everyone,

I’m learning about Delta Lake performance and came across data skipping.

I understand that it can help Databricks avoid reading unnecessary data when running queries, but I’d like to understand its purpose more clearly.

For example, if an orders table contains millions of records and I query only orders from a particular date, how can data skipping help reduce the amount of data that needs to be read?

What is the main purpose of data skipping, and how does it help improve query performance?

Thanks!

Genie - Cost monitoring and usage

Sanjeeb2024 — Sat, 05 Sep 2026 05:42:31 GMT

Hi Team,

Genie is widely used within our department, and we would like to establish appropriate governance around its usage. Do we have any existing guidelines or recommended practices for this? In particular, we are looking for guidance on:

Cost monitoring: How can we monitor Genie usage, including who is using it and the associated consumption or cost?
Usage limits: Is it possible to set an upper limit or other controls to restrict Genie usage?Any relevant documentation, recommendations, or examples would be appreciated.

Regards - Sanjeeb

Databricks Apps - Deployment process - Need help !!

Sanjeeb2024 — Sat, 05 Sep 2026 05:38:57 GMT

Hi all,

Do we have any guidelines or a CI/CD approach for deploying a Databricks app from one environment (e.g., DEV) to PROD?We are currently facing the following challenge:We created a Databricks app in the DEV environment through the UI. At runtime, the app creates a service principal, to which we granted the required permissions to access Databricks tables. After deploying the code, everything works as expected in DEV.How can we fully automate the deployment process from DEV to PROD? In particular, how can we automate assigning the required permissions to the service principal that is created at runtime in the target environment?Any guidance or examples would be appreciated.

Regards

Who is The Best Cold Email Service Provider in 2026?

ahengzhouking — Fri, 04 Sep 2026 20:38:05 GMT

I have been working in the SMMA field for more than 5 years. So I have very strong experience about that.

I tried many cold email providers; I use more than 10k emails from every provider, but everyone has a common problem. They will now give you a long-time replacement guarantee; they provide a maximum of 30 days replacement with their accounts. and one more big problem: inbox rate not more than 80%.

But a few months ago, I tried someone cold email named XecureShop. They provide cold emails with lifetime guarantees and more than a 90% inbox rate. One more interesting thing: they make cold emails by hand. They're not using any types of bot or software.

Now let's talk about the price. When I bought it, the price was for 1000 emails at $1500.

If anyone wants to contact them, use W,h,at, sA,pp: +1 (765) 819-6127

Does enabling Photon improve performance while lowering compute costs?

neerajdubey_86 — Fri, 04 Sep 2026 18:09:09 GMT

Please refer screen shot and help in clarifying if enabling Photon lead to lower execution costs?

Using Temporary Functions

Marbricks — Fri, 04 Sep 2026 17:31:51 GMT

Hi everyone,

I'm currently doing my professional internship and working on a reporting project that integrates with Databricks through Unity Catalog functions.

The reporting application is external to Databricks and uses Data Sources, Parameters, Result Sets, Grids, Charts, and Report Definitions to build dashboards. Existing reports consume data through functions referenced by a URI pattern similar to:

fn://catalog.schema.function_name@DATABRICKS.UNITYCATALOG
Show more lines

My mentor provided an existing report as a starting point and asked me to use it as a template for the new development. The report is already functional and contains the complete structure, including Data Sources, Parameters, Result Sets, Grids, Charts, and Report Definitions. My task is to understand how those pieces work together and then adapt or replace them to support a different dataset.

The challenge is that the new dataset does not currently have equivalent Unity Catalog functions available.

As guidance, my mentor suggested creating temporary functions first, using a pattern such as:

SQL
1
CREATE OR REPLACE TEMPORARY FUNCTION `catalog.schema.function_name`
2
RETURNS TABLE (...)
3
RETURN ...
4

Show more lines

The idea is to prototype the expected interface, validate the returned schema locally with the reporting application, and later create the permanent Unity Catalog functions if the design works correctly.

My mentor has already given me a clear direction, but as an intern I am still connecting the dots and trying to understand the complete flow from temporary functions, to production Unity Catalog functions, to the final report displayed in the application.

Has anyone worked on a similar integration between an external dashboard/reporting application and Databricks?

Any guidance, recommendations, or examples would be greatly appreciated.

I'm still pretty new to this area, so honestly I'm feeling a bit lost and trying to understand the correct path before I spend time going in the wrong direction 😅.

Thanks in advance! 👍

Lakeflow connect Ingestion pipeline notification for gateway pipeline

srikanthp24 — Fri, 04 Sep 2026 17:30:21 GMT

Hi Guys,

As you guys know that when we are building lake flow connect ingestion pipeline in UI. The pipeline consist both gateway pipeline and ingestion pipeline together. We have notification for the ingestion pipeline but not for gateway pipeline. I would like to track or keep an on gateway pipeline, If it fail I need to get the email alert. How we can achiever this. Can anybody face similer problem. the UI looks like as below ss

Clarification on Automating Serverless Compute Permissions via SDK/API

saicharan1 — Fri, 04 Sep 2026 15:10:13 GMT

Hi Team,

I'm trying to automate granting Can Use permissions on Default Interactive Compute for workspace groups across multiple Databricks workspaces. While reviewing the Serverless Compute access control documentation and the Access Control Rule Set APIs, I noticed that the examples require a resource name for get_rule_set() and update_rule_set(), but I couldn't find any documentation that explains how to discover the correct resource name for Default Interactive Compute or Default Automated Compute.

I also noticed that the Databricks Python SDK exposes WorkspaceClient.account_access_control_proxy with methods such as get_rule_set(), update_rule_set(), and get_assignable_roles_for_resource(). However, without knowing the resource name, it's unclear how these APIs can be used programmatically.

Could someone clarify:

How can we discover the correct resource name for Serverless Compute objects programmatically?
Is there an API to list available Serverless Compute resources and their associated rule sets?
Is Serverless Compute access control fundamentally a workspace-level permission model or an account-level one? The UI suggests each workspace has its own Default Interactive/Automated Compute objects, while the APIs appear to be account-focused.
Is account_access_control_proxy the recommended SDK for this use case, or is there another supported SDK/API specifically intended for managing Serverless Compute permissions?
Our goal is to automate permission assignment for groups across many workspaces, so any guidance on the correct endpoint, resource naming convention, or recommended SDK approach would be greatly appreciated.
Thanks!

Serverless compute access control - Azure Databricks

fastai import in databricks broken

barnabywalker — Fri, 04 Sep 2026 14:47:47 GMT

Has anyone else had a problem today (2026-09-04) with imports from fastai?

The error message points to a problem with the underlying fastcore package:

AttributeError: 'Function' object attribute '__doc__' is read-only

I'm getting the same error using fastai versions 2.8.5 and 2.8.8.

The only change I can see is in the compute event logs, my 'Compute is using release 18.3.6' compared to 18.3.5 yesterday.

Genie Agent content search on a managed volume fails with "The request was invalid"

mramanindia — Fri, 04 Sep 2026 14:08:06 GMT

Hi all,

I am trying to enable content search on a Unity Catalog managed volume so a Genie Agent can answer questions over PDFs. The Enable action fails every time with:

"Failed to update content search - The request was invalid."

Environment: AWS, eu-west-2 (London), serverless workspace on Default Storage, pay-as-you-go. This started as a trial and the failure is identical before and after adding a payment method.

What I have verified:

- Preview "Analyze Files in Volumes with Genie Agents" is enabled.

- The volume is a managed volume. Catalog binding is "All workspaces have access" (OPEN). Reproduced on two catalogs, including a freshly created one.

- I have CAN MANAGE on the volume.

- ai_parse_document() runs successfully on the PDFs in the volume.

- Foundation Model APIs work (ai_query succeeds).

- Lakebase is available in the region.

- I can create a Vector Search endpoint via the API and it comes ONLINE.

- Files are 4 PDFs, each under 10 MB.

Without content search the agent does not retrieve the files ("no relevant content from the reports directory"), so document and hybrid questions are not possible.

Questions:

1. Is content search supported on Default Storage / serverless-only workspaces? It is not listed under the Default Storage limitations, but it is the one variable I cannot rule out.

2. Is there a way to get a more detailed error than "The request was invalid"? No public API seems to expose this.

Happy to share the request ID from the browser network log if that helps.

Thanks!

VS Code and Connect - Manual Environment Setup

spoltier — Fri, 04 Sep 2026 09:05:44 GMT

The current version of the extension forces usage of certain python packaging / project tools. I used to be able to use the extension with my project which uses pixi, by setting up the dependencies myself in pyproject.toml / pixi.toml.

The current version of the extension ignores that setup completely and starts setting up a project based on uv. This basically makes the connect functionality unusable for my project.

I would like to be able to select the kernel / python environment myself, without the extension making a bunch of changes to my project configuration.

Error While Running Workloads on Databricks Free Edition – Serverless Compute

Bharat_Kumar_AV — Fri, 04 Sep 2026 06:18:44 GMT

Hi Team,

I am using the Databricks Free Edition and I am facing an issue while running workloads using Serverless Compute.

When I try to run the workload, I receive the following error:

“An error occurred while trying to provision serverless compute. Please try again or contact support.”

I have tried running the workload again, but the issue persists.

Could you please check if there is any issue with the serverless compute provisioning for my workspace or if any configuration/action is required from my side?

What is a Warehouse in Databricks SQL?

gowri_databrick — Fri, 04 Sep 2026 04:36:00 GMT

Hi everyone,

I’m learning Databricks SQL and came across SQL warehouses.

I understand that a SQL warehouse provides compute resources for running SQL queries, but I’m not clear about how it is different from other compute options in Databricks.

When should we use a SQL warehouse, and what are the main things to consider when choosing one?

Thanks!