I'm looking to migrate onto unity catalog but a number of my data ingestion notebooks throw a securityexception/whitelist errors for numerous spark. functions

Is there some configuration setting I need to enable to whitelist the spark.* methods/functions?

I know its because I'm using 'shared' access mode. I've always run 'no isolation shared' clusters before with external tables when using hive metastore

I use externally managed tables and use spark.catalog to check if a table exists before I create it. This is failing with the whitelist error. I can refactor that check to use the information_schema columns I guess?

But any tips on how to refactor this?

I have multiple tsvs which have free text comments at the top of the file. I need to skip n lines and process the rest

    row_rdd = spark.sparkContext \
        .textFile(sourceFilePath) \
        .zipWithIndex() \
        .filter(lambda row: row[1] >= n_skip_rows) \
        .map(lambda row: row[0])
    df = spark.read.csv(row_rdd,sep='\t',header="true",inferSchema="true")

I also need to process vcfs using the glow library - this doesn't work either

Are there any docs on what Single user access mode actually is? Is it like its running using someone's credentials as a service account? Can other users connect to it using odbc/jdbc and an access token? Or is it a personal compute which only allows one connection?