Vacuuming clones in USER_ISOLATION mode and ThreadPool Executor

radhag — Fri, 12 Sep 2025 01:01:07 GMT

Hello,

I run my process on a shared, interactive cluster (data security mode: USER_ISOLATION).

I run operations on multiple tables having each of them as a separate thread, pseudo-code :

try:

truncate target_table

vacuum target_table (retain 0 hours with duration check spark conf set to 'false')

drop target_table

create table target_table deep/shallow clone source_table timestamp as of '<timestamp>'

I don't have MODIFY privilege on the source_table but I have every other permission, I have ALL PRIVILEGES and MANAGE on target_table.

The only operation when running via ThreadPoolExecutor that fails is vacuuming due to lack of MODIFY permission on source_table according to the output of the error - just as if I used the SINGLE_USER mode (a dedicated cluster) but I'm using USER_ISOLATION.

What might be interesting is that in the same run when I print the vacuum statement that failed and pass it in a new cell in the same notebook on the same cluster and run it - it would run successfully.

Is it a bug?

Regards,

Radoslaw

Re: Vacuuming clones in USER_ISOLATION mode and ThreadPool Executor

mark_ott — Fri, 19 Sep 2025 12:12:41 GMT

This is most likely a bug. It certainly is unexpected and should be reported to Databricks support or your platform administrator for clarification and remediation.

As a temporary workaround, running vacuum commands outside of threaded contexts or scheduling them sequentially may avoid the issue until a fix is implemented.

topic Vacuuming clones in USER_ISOLATION mode and ThreadPool Executor in Data Engineering

Vacuuming clones in USER_ISOLATION mode and ThreadPool Executor

Re: Vacuuming clones in USER_ISOLATION mode and ThreadPool Executor