Read CSV file using SQL

Rami2023 — Thu, 20 Apr 2023 14:49:45 GMT

Hi, I am trying to reverse engineer to get to the source file for a table. Looking at the query history, I came across SQL string which loads data from file to table, however the code looks little mystery to me. I haven't come across idbfs, Can somebody help me understanding.

COPY INTO 'tablename_temp' FROM (

SELECT

try_cast(`_type` AS string) `_type`,

try_cast(`ID` AS double) `ID`,

try_cast(`Number` AS string) `Number`,

try_cast(`Name` AS string) `Name`,

try_cast(`Type` AS string) `Type`

FROM

'idbfs:/2023-03-29/10/_fe93bfcf-4ad6-4e14-b2d7-9811ddbc0c7d'

) FILEFORMAT = CSV FORMAT_OPTIONS (

'inferSchema' = 'true',

'multiLine' = 'false',

'header' = 'true',

'delimiter' = ',',

'escape' = '"',

'mergeSchema' = 'true',

'recursiveFileLookup' = 'true'

) COPY_OPTIONS ('overwrite' = 'true', 'overwriteSchema' = 'true')

)

Re: Read CSV file using SQL

Rami2023 — Fri, 21 Apr 2023 07:41:48 GMT

Hi Kaniz,

Thanks for your helpful reply. I guessed the same. I have never come across idbfs file system. Is this something new with Databricks ? Any reference would be helpful to get to the source.

topic Re: Read CSV file using SQL in Data Engineering

Read CSV file using SQL

Re: Read CSV file using SQL