- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2024 05:25 AM - edited 01-22-2024 05:27 AM
I want to create the columns with datatype struct and array of struct datatype in the DLT live tables, will it be possible, if possible could you share the sample for the same.
Thanks.
- Labels:
-
Delta Lake
-
Workflows
Accepted Solutions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2024 06:16 AM
Hi @Data_Engineer3 , In DLT, we don't create empty tables to begin with but we create the tables by reading the source. For ex:-
CREATE STREAMING LIVE TABLE customers
AS SELECT * FROM cloud_files("/databricks-datasets/retail-org/customers/", "csv");
If your source data has the struct and array columns, your DLT tables will also have these columns as a result.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2024 09:57 AM - edited 01-22-2024 09:58 AM
Hi @Lakshay
Thanks for reply,
Yes, my transformation query(i.e dlt view) will have the columns with type struct field like(Column_name) below
Column_O STRING,
Column_name STRUCT< vaccines
ARRAY<
STRUCT< vaccineId STRING, date STRING > >
satisfiedBy ARRAY< STRUCT< vaccineId STRING, orderId STRING, orderItemId STRING >>
serviceName STRING,
dueDate STRING,
completedAt STRING,
state ARRAY< STRUCT<id STRING, name STRING>>,
nextDueDate STRING
>,
Column_name_2 STRING,
.is it possible to create the column with above structure, basically it is combination of both structure and array of structure.
Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-23-2024 03:14 AM
You should be able to access the logs from the right side panel under Update details. Please refer the screenshot below
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2024 06:16 AM
Hi @Data_Engineer3 , In DLT, we don't create empty tables to begin with but we create the tables by reading the source. For ex:-
CREATE STREAMING LIVE TABLE customers
AS SELECT * FROM cloud_files("/databricks-datasets/retail-org/customers/", "csv");
If your source data has the struct and array columns, your DLT tables will also have these columns as a result.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2024 09:57 AM - edited 01-22-2024 09:58 AM
Hi @Lakshay
Thanks for reply,
Yes, my transformation query(i.e dlt view) will have the columns with type struct field like(Column_name) below
Column_O STRING,
Column_name STRUCT< vaccines
ARRAY<
STRUCT< vaccineId STRING, date STRING > >
satisfiedBy ARRAY< STRUCT< vaccineId STRING, orderId STRING, orderItemId STRING >>
serviceName STRING,
dueDate STRING,
completedAt STRING,
state ARRAY< STRUCT<id STRING, name STRING>>,
nextDueDate STRING
>,
Column_name_2 STRING,
.is it possible to create the column with above structure, basically it is combination of both structure and array of structure.
Thanks
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2024 11:43 AM
I dont think there should an issue with it
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-22-2024 11:16 PM
I have created DLT live tables pipeline, In Job UI, i can able to see only steps and if any failure happened it show only error at that stage.
But if i use any log using print, it doesn't show the logs in the console or any where. how can i see the logs associated with the pipelines. Is there any way to check the logs in DLT flow for development??How do I see the logs of the running DLT pipeline
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-23-2024 03:14 AM
You should be able to access the logs from the right side panel under Update details. Please refer the screenshot below