topic Re: tqdm progressbar in Databricks jobs in Data Engineering

tqdm progressbar in Databricks jobs

majo2 — Tue, 20 Aug 2024 16:37:33 GMT

Hi,

I'm using Databricks workflows to run a train job using `pytorch` + `lightning`. `lightning` has a built in progressbar built on `tqdm` that tracks the progress. It works OK in when I run the notebook outside of a workflow. But when I try to run notebook as a part of a workflow no progressbar is displayed.

Is this some limitation of the workflows or am I hitting some kind of bug?

I'm on Databricks runtime for ML 15.3 with `tqdm==4.65.0

Re: tqdm progressbar in Databricks jobs

spatel — Sun, 29 Sep 2024 14:46:10 GMT

I'm having the same issue. I even tried following

from tqdm.autonotebook import tqdm

but when I do this

for row in tqdm(df.itertuples(index=False)):

I don't see the progress bar in Databricks workflow notebook.

Re: tqdm progressbar in Databricks jobs

ludovicc — Fri, 24 Jan 2025 20:58:57 GMT

I have found that only progressbar2 can work in both interactive notebooks and workflow notebooks. It's limited, but better than nothing. Tqdm is broken in workflows.