I am noticing a difference between using autoloader in an interactive notebook vs using it in a Spark Declarative Pipeline (DLT Pipeline). This issue seems to be very similar to this other unanswered post from a few years ago. Bug report: the delimiter option does not work when run on DLT 
The CSV I am trying to ingest uses the pipe character (|) as its separator.

```

    df = (spark.readStream
        .format("cloudFiles")
        .option("cloudFiles.format", "csv")
        .option("delimiter", "|")
        .option("cloudFiles.schemaLocation", f"/Volumes/{location}/schema")
        .load(f"/Volumes/{path to csv}")
        .writeStream
        .option("checkpointLocation", f"/Volumes/{location}/checkpoint")
        .trigger(availableNow=True)
        .toTable(f"{table_location}")
    )
```
^ running this in a notebook serverless has no issues.

``` python
@Dlt.table(
        name=f"{bronze_raw_schema}.{table_name}",
    )
def load_table():
      return spark.readStream.format("cloudFiles")
        .format("cloudFiles")
        .option("cloudFiles.format", "csv")
        .option("delimiter", "|")
        .option("header", "true")
        .option("cloudFiles.includeExistingFiles", "true")
        .load(f"/Volumes/{path to csv}")
```

When I run the above in a SDP (DLT) pipeline I get the following error. Which appears to be failing to delimit on the pipe (|) character and is thus hitting a length limit on the header column names.

com.databricks.sql.managedcatalog.UnityCatalogServiceException: [ErrorClass=INVALID_PARAMETER_VALUE] Invalid input: RPC UpdateTable Field managedcatalog.ColumnInfo.name: At columns.0: name "HiPortSystemCode|HiportCode|CashAccountCode|ValuationDate|ContractDate|CashTransactionId|DeletedIndicator|CashTransactionTypeCode|AccountingDate|EntryDate|SettlementDate|OriginalAccountingDate|SecurityCode|AnalysisTypeCode|TransferCashAccountCode|Transact..." too long. Maximum length is 255 characters.