Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-16-2025 08:25 AM
@Costas96 I would recommend to verify the sql behavior in a notebook/sql editor.
# SQL query with proper comma separation between columns
query = """
SELECT
a.column_a as id_column,
a.column_b as val_column
FROM
catalog_a.schema_a.table_a a
UNION ALL
SELECT
b.column_a as id_column,
b.column_b as val_column
FROM
catalog_b.shema_b.table_b b"""
# Define the Delta Live Table
@dlt.table
def dim_ship():
return spark.sql(query)
# Optional: Verify the output
df = spark.sql(query)
# Check schema
print("Schema:")
df.printSchema()
# Preview data
print("\nData Preview:")
df.show(5)