Vacuuming clones in USER_ISOLATION mode and ThreadPool Executor

[radhag]

Hello,

I run my process on a shared, interactive cluster (data security mode: USER_ISOLATION).

I run operations on multiple tables having each of them as a separate thread, pseudo-code :

try: 

truncate target_table

vacuum target_table (retain 0 hours with duration check spark conf set to 'false') 

drop target_table

create table target_table deep/shallow clone source_table timestamp as of '<timestamp>'

I don't have MODIFY privilege on the source_table but  I have every other permission, I have ALL PRIVILEGES and MANAGE on target_table.

The only operation when running via ThreadPoolExecutor that fails is vacuuming due to lack of MODIFY permission on source_table according to the output of the error - just as if I used the SINGLE_USER mode (a dedicated cluster) but I'm using USER_ISOLATION.

What might be interesting is that in the same run when I print  the vacuum statement that failed and pass it in a new cell in the same notebook on the same cluster and run it - it would run successfully.

Is it a bug?

Regards,

Radoslaw

[mark_ott]

This is most likely a bug. It certainly is unexpected and should be reported to Databricks support or your platform administrator for clarification and remediation.

• As a temporary workaround, running vacuum commands outside of threaded contexts or scheduling them sequentially may avoid the issue until a fix is implemented.