Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-16-2025 04:46 AM - edited 01-16-2025 05:02 AM
Hello everyone. I am new to DLT and I am trying to practice with it by doing some basic ingestions. I have a query like the following where I am getting data from two tables using UNION. I have noticed that everything gets ingested at the first column as a comma separated string. In my pipeline I am executing somethin like the following. Any suggestions would be appreciated. Cheers!
query = """
SELECT
a.column_a as id_column
a.column_b as val_column
FROM
catalog_a.schema_a.table_a a
UNION ALL
SELECT
b.column_a as id_column
b.column_b as val_column
FROM
catalog_b.shema_b.table_b b"""
@dlt.table
def dim_ship():
return spark.sql(query)