Map("skipRows", "1") ignored during autoloader process. Something wrong with the format?

Chris_Shehu — Tue, 20 Dec 2022 21:49:08 GMT

I've tried multiple variations of the following code. It seems like the map parameters are being completely ignored.

CREATE LIVE TABLE a_raw2
TBLPROPERTIES ("quality" = "bronze")
AS SELECT * FROM cloud_files("dbfs:/mnt/c-raw/a/c_medcheck_export*.csv", "csv", map("skipRows","1" , "header", "true", "cloudFiles.inferColumnTypes", "true"))

com.databricks.sql.transaction.tahoe.DeltaAnalysisException: Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema.

Please upgrade your Delta table to reader version 2 and writer version 5

and change the column mapping mode to 'name' mapping. You can use the following command:

ALTER TABLE <table_name> SET TBLPROPERTIES (

'delta.columnMapping.mode' = 'name',

'delta.minReaderVersion' = '2',

'delta.minWriterVersion' = '5')

Refer to table versioning at https://docs.microsoft.com/azure/databricks/delta/versioning

org.apache.spark.sql.AnalysisException: Column name "Duration (in seconds)" contains invalid character(s). Please use alias to rename it.

Re: Map("skipRows", "1") ignored during autoloader process. Something wrong with the format?

jose_gonzalez — Wed, 28 Dec 2022 00:04:09 GMT

skipRows was added in DBR 11.1 -- what DBR is your DLT pipeline on?

Re: Map("skipRows", "1") ignored during autoloader process. Something wrong with the format?

Chris_Shehu — Thu, 29 Dec 2022 16:14:09 GMT

Good morning, I'm using 12.0.

topic Re: Map("skipRows", "1") ignored during autoloader process. Something wrong with the format? in Data Engineering

Map("skipRows", "1") ignored during autoloader process. Something wrong with the format?

Re: Map("skipRows", "1") ignored during autoloader process. Something wrong with the format?

Re: Map("skipRows", "1") ignored during autoloader process. Something wrong with the format?