I'm getting an error when I try to create a DLT Pipeline from a bunch of third-party app-usage data we have. Here's the error message:
Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema.
Please upgrade your Delta table to reader version 2 and writer version 5
and change the column mapping mode to 'name' mapping. You can use the following command:
ALTER TABLE <table_name> SET TBLPROPERTIES (
'delta.columnMapping.mode' = 'name',
'delta.minReaderVersion' = '2',
'delta.minWriterVersion' = '5')
So, I added the properties to my table definition, and I'm still getting the error. What am I doing wrong? Here's the table definition:
CREATE STREAMING LIVE TABLE clevertap_analytics_bronze
COMMENT "App usage data from CleverTap"
TBLPROPERTIES ("myCustomPipeline.quality" = "bronze",
"delta.columnMapping.mode" = "name",
"delta.minReaderVersion" = "2",
"delta.minWriterVersion" = "5"
)
AS
SELECT
*
FROM
cloud_files(
-- REPLACE THE BELOW LINE WITH THE EXACT S3 LOCATION WHERE YOU DATA LIVES
"s3://clevertap-analytics/",
"json",
-- CHANGE THE FOLLOWING TO "false" IF THE CSV FILE(s) DO NOT INCLUDE A HEADER
map(
"header", "true",
"cloudFiles.inferColumnTypes", "true",
"cloudFiles.schemaEvolutionMode", "rescue",
"rescuedDataColumn", "rescue_col"
)
);
