Setting up my first DLT Pipeline with 3rd party JSON data

thains
New Contributor III

I'm getting an error when I try to create a DLT Pipeline from a bunch of third-party app-usage data we have. Here's the error message:

Found invalid character(s) among ' ,;{}()\n\t=' in the column names of your schema.

Please upgrade your Delta table to reader version 2 and writer version 5

and change the column mapping mode to 'name' mapping. You can use the following command:

ALTER TABLE <table_name> SET TBLPROPERTIES (

'delta.columnMapping.mode' = 'name',

'delta.minReaderVersion' = '2',

'delta.minWriterVersion' = '5')

So, I added the properties to my table definition, and I'm still getting the error. What am I doing wrong? Here's the table definition:

CREATE STREAMING LIVE TABLE clevertap_analytics_bronze 

 COMMENT "App usage data from CleverTap" 

 TBLPROPERTIES ("myCustomPipeline.quality" = "bronze",

  "delta.columnMapping.mode" = "name",

  "delta.minReaderVersion" = "2",

  "delta.minWriterVersion" = "5"

AS

SELECT

 *

FROM

 cloud_files(

  -- REPLACE THE BELOW LINE WITH THE EXACT S3 LOCATION WHERE YOU DATA LIVES

  "s3://clevertap-analytics/",

  "json",

 -- CHANGE THE FOLLOWING TO "false" IF THE CSV FILE(s) DO NOT INCLUDE A HEADER

  map(

   "header", "true",

   "cloudFiles.inferColumnTypes", "true",

   "cloudFiles.schemaEvolutionMode", "rescue",

   "rescuedDataColumn", "rescue_col"

  )

 );