To access init script on ADLS the Hadoop API is used so you need to provide correct Spark configuration using properties prefixed by spark.hadoop. For example, if you use service principal, you need following properties (taken from example for Terraform😞

spark.hadoop.fs.azure.account.auth.type OAuth
spark.hadoop.fs.azure.account.oauth.provider.type org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
spark.hadoop.fs.azure.account.oauth2.client.endpoint https://login.microsoftonline.com/<azure_tenant_id>/oauth2/token
spark.hadoop.fs.azure.account.oauth2.client.id <azure_client_id>
spark.hadoop.fs.azure.account.oauth2.client.secret {{secrets/<azure_client_secret_secret_scope>/<azure_client_secret_secret_key>}}