Map("skipRows", "1") ignored during autoloader process. Something wrong with the format?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-20-2022 01:49 PM
I've tried multiple variations of the following code. It seems like the map parameters are being completely ignored.
CREATE LIVE TABLE a_raw2
TBLPROPERTIES ("quality" = "bronze")
AS SELECT * FROM cloud_files("dbfs:/mnt/c-raw/a/c_medcheck_export*.csv", "csv", map("skipRows","1" , "header", "true", "cloudFiles.inferColumnTypes", "true"))
com.databricks.sql.transaction.tahoe.DeltaAnalysisException: Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema.
Please upgrade your Delta table to reader version 2 and writer version 5
and change the column mapping mode to 'name' mapping. You can use the following command:
ALTER TABLE <table_name> SET TBLPROPERTIES (
'delta.columnMapping.mode' = 'name',
'delta.minReaderVersion' = '2',
'delta.minWriterVersion' = '5')
Refer to table versioning at https://docs.microsoft.com/azure/databricks/delta/versioning
org.apache.spark.sql.AnalysisException: Column name "Duration (in seconds)" contains invalid character(s). Please use alias to rename it.
- Labels:
-
Autoloader
-
Delta
-
DLT
-
Map
-
SQL
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-27-2022 04:04 PM
skipRows was added in DBR 11.1 -- what DBR is your DLT pipeline on?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
12-29-2022 08:14 AM
Good morning, I'm using 12.0.

