I've tried multiple variations of the following code. It seems like the map parameters are being completely ignored.
CREATE LIVE TABLE a_raw2
TBLPROPERTIES ("quality" = "bronze")
AS SELECT * FROM cloud_files("dbfs:/mnt/c-raw/a/c_medcheck_export*.csv", "csv", map("skipRows","1" , "header", "true", "cloudFiles.inferColumnTypes", "true"))
com.databricks.sql.transaction.tahoe.DeltaAnalysisException: Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema.
Please upgrade your Delta table to reader version 2 and writer version 5
and change the column mapping mode to 'name' mapping. You can use the following command:
ALTER TABLE <table_name> SET TBLPROPERTIES (
'delta.columnMapping.mode' = 'name',
'delta.minReaderVersion' = '2',
'delta.minWriterVersion' = '5')
Refer to table versioning at https://docs.microsoft.com/azure/databricks/delta/versioning
org.apache.spark.sql.AnalysisException: Column name "Duration (in seconds)" contains invalid character(s). Please use alias to rename it.