Using a cluster in serverless mode, three tables are joined and the data frame is written as follows
df.write.mode('append').saveAsTable('table name')
and shema is below
- date string (ymd format)
- id bigint
- value string
- partition by date
After about one minute of execution as a job, the profiler stops progressing and does not return any response.
When I canceled the job, the progress of the profiler was updated with progress, and the tree looked as if it stopped writing to the delta table.
When I reduced the amount of data, it succeeded, but when I rerun the job, the same problem occurs. The same problem occurs when the table is re-created and re-run.
Also, I comment out the write process and modify the display to show the result, the result comes back in about 1 minute.
The select seems to work fine, but only the write is causing this problem.
What kind of analysis should i do?
