We are getting
\u318a (ㆊ)
separated csv file. We want to create unmanaged table in databricks, Here is the table creation script.
create table IF NOT EXISTS db_test_raw.t_data_otc_poc
(`caseidt` String,
`worktype` String,
`doctyp` String,
`brand` String,
`reqemailid` String,
`subprocess` String,
`accountname` String,
`location` String,
`lineitems` String,
`emailsubject` String,
`createddate` string,
`process` String,
`archivalbatchid` String,
`createddt` String,
`customername` String,
`invoicetype` String,
`month` String,
`payernumber` String,
`sapaccountnumber` String,SOURCE_BUSINESS_DATE Date ) USING
CSV OPTIONS (header 'true',encoding 'UTF-8',quote '"', escape '"',delimiter '\u318a', path
'abfss://xxxx@yyyyy.dfs.core.windows.net/Raw/OPERATIONS/BUSINESSSERVICES/***/xx_DATA_OTC')
PARTITIONED BY (SOURCE_BUSINESS_DATE )
The table created successfully in databricks.
While checking (
describe table extended db_test_raw.t_data_otc_poc
), we found storage properties as [encoding=UTF-8, quote=", escape=", header=true, delimiter=?] .The delimiter got changed.
Can you please let us know what went wrong here?
Data is also loaded into first columns and value for the rest of the column is null
