just curious, will using the same job cluster within the same workflow work ? Theoretically they should work. If it is across jobs with different job clusters they may not work and persistent tables are the solution. you could drop the table at the end of the flow with a generic task.