Hi,
sometime I notice that running a query takes too long - even simple queries - and next time when I run same query it runs much faster. I have cluster running (DBR 10.4 LTS • 5 workers) and it has constantly several workers.
An Example of query is simple select on table which I truncated before, so I know it is empty, and I do something like:
#
df = spark.sql(
f"""
select count(*) from table_name
"""
)
display(df)
First time it took 1.3 minutes and running it again took 0.6 sec.
It seems to happen quite often, as if waiting for something to start even though it should be started and running.
Do you have some explanation for this behavior and how I can help it?
Thank you!