Need to define the struct and array of struct field colum in the delta live table(dlt) in databrick.

[Data_Engineer3]

I want to create the columns with datatype struct and array of struct datatype in the DLT live tables, will it be possible, if possible could you share the sample for the same.

Thanks.

[Lakshay · Accepted solution]

Hi @Data_Engineer3 , In DLT, we don't create empty tables to begin with but we create the tables by reading the source. For ex:-

CREATE STREAMING LIVE TABLE customers
AS SELECT * FROM cloud_files("/databricks-datasets/retail-org/customers/", "csv");

 If your source data has the struct and array columns, your DLT tables will also have these columns as a result.  

[Data_Engineer3 · Accepted solution]

Hi @Lakshay 

Thanks for reply,

Yes, my transformation query(i.e dlt view) will have the columns with type struct field like(Column_name) below

Column_O STRING,

Column_name STRUCT< vaccines
ARRAY<
STRUCT< vaccineId STRING, date STRING > >
satisfiedBy ARRAY< STRUCT< vaccineId STRING, orderId STRING, orderItemId STRING >>
serviceName STRING,
dueDate STRING,
completedAt STRING,
state ARRAY< STRUCT<id STRING, name STRING>>,
nextDueDate STRING
>,

Column_name_2   STRING,

.is it possible to create the column with above structure, basically it is combination of both structure and array of structure.

  Thanks

[Lakshay]

I dont think there should an issue with it

[Data_Engineer3]

I have created DLT live tables pipeline, In Job UI, i can able to see only steps and if any failure happened it show only error at that stage.
But if i use any log using print, it doesn't show the logs in the console or any where. how can i see the logs associated with the pipelines. Is there any way to check the logs in DLT flow for development??How do I see the logs of the running DLT pipeline

[Lakshay · Accepted solution]

You should be able to access the logs from the right side panel under Update details. Please refer the screenshot below