Delta Live Tables: Creating table with spark.sql and everything gets ingested at the first column

Costas96
New Contributor III

Hello everyone. I am new to DLT and I am trying to practice with it by doing some basic ingestions. I have a query like the following where I am getting data from two tables using UNION. I have noticed that everything gets ingested at the first column as a comma separated string. In my pipeline I am executing somethin like the following. Any suggestions would be appreciated. Cheers!

 

 

 

 

query = """
SELECT
a.column_a as id_column
a.column_b as val_column
FROM 
    catalog_a.schema_a.table_a a

UNION ALL

SELECT
b.column_a as id_column
b.column_b as val_column
FROM 
    catalog_b.shema_b.table_b b"""


@dlt.table
def dim_ship():
    return spark.sql(query)