- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
11-15-2024 10:49 AM - edited 11-15-2024 10:53 AM
Hi @ChristianRRL ,
(A) You're not missing anything, there's no such an option as of today for SQL API.
(B) It would be much better for you to just use pyspark, but if you have to stick to just SQL API you can use following aproach. Define your schema using DDL String and store that in simple text file.
Then you can prepare variable called schema. Read into that variable content of schema text file and as a last step, pass schema variable to read_files.
%sql
DECLARE OR REPLACE VARIABLE schema STRING;
select value as schema
from read_files(
'dbfs:/FileStore/schema.txt'
,format => 'text'
)
select *,_metadata
from read_files(
'dbfs:/FileStore/Test.csv'
,format => 'csv'
,schema => schema
,rescuedDataColumn => '_rescued_data'
)
;
Here is an example content of schema file. It can be arbitraly complex, but for the sake of an example I only attached simple data types. Storing it as DDL String in text file allows you to avoid using udf. The drawback is, that you lose some readability when the schema is defined using ddl string.