Databricks Community

bcsalay · 12-10-2024

Hi,I'm encountering an issue in a pyspark code, where I'm calculating certain information monthly in a loop. The flow is pretty much as:Read input and create/read intermediate parquet files,Upsert records in intermediate parquet files with the monthl...

bcsalay · 12-17-2024

Hi @JacekLaskowski thank you for your response. No it is not a catalog table and not accessed/used by another job. I tried to explain above what the code does operationally, giving some more context: it is a development code, which processes historic...

bcsalay · 12-17-2024

Hi @MuthuLakshmi thank you for your response. No I don't use df.cache() anywhere in the code. Yet I tried uncaching intermediate table which is read and updated within the loop, but it didn't help:spark.catalog.uncacheTable("IM_AWL_Products")I don't ...

Databricks Community

User Stats

User Activity

Random failure in the loop in pyspark

Re: Random failure in the loop in pyspark

Re: Random failure in the loop in pyspark