I am migrating my Data Lake to use Unity Catalog. However, this comes with changes to the clusters. I have tried a few options, but it seems rather complex than it should be.

I need to create a cluster used by ADF that is Unity Enabled that can install a JAR. From my testing, a shared cluster cannot use dbutils which I need to pass parameters (i.e tablename). It also does not allow libraries / JAR to be installed.

A single user interactive cluster seems like the right approach. However, I am not able to add the ADF service principal as a user.

A job cluster works. But I have many pipelines and databricks notebook jobs that runs daily. So it seems rather excessive to kickstart X clusters when one or two interactive cluster can be used

What is the right approach here for creating a cluster for ADF that is UC enabled, allows dbutils and can have a JAR installed on it?

Is running more Job clusters more expensive than one interactive all-purpose one?