aladda
Databricks Employee
Databricks Employee

Here's an example for predefined schema

  1. Using COPY INTO with a predefined table schema – Trick here is to CAST the CSV dataset into your desired schema in the select statement of COPY INTO. Example below

%sql 

CREATE OR REPLACE TABLE copy_into_bronze_test 

(id STRING, reading_time TIMESTAMP, device_type STRING, device_id STRING, device_operational_status STRING, reading_1 double, reading_2 double, reading_3 double)

USING DELTA

 

%sql 

COPY INTO copy_into_bronze_test

FROM

 (

   SELECT

     _c0 as ID,

     CAST (_c1 AS timestamp) AS reading_time,

     _c2 as device_type,

     _c3 as device_id,

     _c4 as device_operational_status,

     CAST(_c5 as double) as reading_1,

     CAST(_c6 as double) as reading_2,

     CAST(_c7 as double) as reading_3

  

FROM

 'dbfs:/FileStore/flight/anand.ladda/*_no_headers.csv'

 )

FILEFORMAT = CSV FORMAT_OPTIONS('sep' = ',','inferSchema' = 'false','header' = 'false')

View solution in original post