cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Map("skipRows", "1") ignored during autoloader process. Something wrong with the format?

Chris_Shehu
Valued Contributor III

I've tried multiple variations of the following code. It seems like the map parameters are being completely ignored.

CREATE LIVE TABLE a_raw2
TBLPROPERTIES ("quality" = "bronze")
AS SELECT * FROM cloud_files("dbfs:/mnt/c-raw/a/c_medcheck_export*.csv", "csv", map("skipRows","1" , "header", "true", "cloudFiles.inferColumnTypes", "true")) 

com.databricks.sql.transaction.tahoe.DeltaAnalysisException: Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema.

Please upgrade your Delta table to reader version 2 and writer version 5

and change the column mapping mode to 'name' mapping. You can use the following command:

ALTER TABLE <table_name> SET TBLPROPERTIES (

'delta.columnMapping.mode' = 'name',

'delta.minReaderVersion' = '2',

'delta.minWriterVersion' = '5')

Refer to table versioning at https://docs.microsoft.com/azure/databricks/delta/versioning

org.apache.spark.sql.AnalysisException: Column name "Duration (in seconds)" contains invalid character(s). Please use alias to rename it.

2 REPLIES 2

jose_gonzalez
Moderator
Moderator

skipRows was added in DBR 11.1 -- what DBR is your DLT pipeline on?

Chris_Shehu
Valued Contributor III

Good morning, I'm using 12.0.

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.