topic Re: Need to define the struct and array of struct field colum in the delta live table(dlt) in databr in Data Engineering

Need to define the struct and array of struct field colum in the delta live table(dlt) in databrick.

Data_Engineer3 — Mon, 22 Jan 2024 13:27:27 GMT

I want to create the columns with datatype struct and array of struct datatype in the DLT live tables, will it be possible, if possible could you share the sample for the same.

Thanks.

Re: Need to define the struct and array of struct field colum in the delta live table(dlt) in databr

Lakshay — Mon, 22 Jan 2024 14:16:32 GMT

Hi @Data_Engineer3 , In DLT, we don't create empty tables to begin with but we create the tables by reading the source. For ex:-

CREATE STREAMING LIVE TABLE customers
AS SELECT * FROM cloud_files("/databricks-datasets/retail-org/customers/", "csv");

If your source data has the struct and array columns, your DLT tables will also have these columns as a result.

Re: Need to define the struct and array of struct field colum in the delta live table(dlt) in databr

Data_Engineer3 — Mon, 22 Jan 2024 17:58:13 GMT

Hi @Lakshay

Thanks for reply,

Yes, my transformation query(i.e dlt view) will have the columns with type struct field like(Column_name) below

Column_O STRING,

Column_name STRUCT< vaccines
ARRAY<
STRUCT< vaccineId STRING, date STRING > >
satisfiedBy ARRAY< STRUCT< vaccineId STRING, orderId STRING, orderItemId STRING >>
serviceName STRING,
dueDate STRING,
completedAt STRING,
state ARRAY< STRUCT<id STRING, name STRING>>,
nextDueDate STRING
>,

Column_name_2 STRING,

.is it possible to create the column with above structure, basically it is combination of both structure and array of structure.

Thanks

Re: Need to define the struct and array of struct field colum in the delta live table(dlt) in databr

Lakshay — Mon, 22 Jan 2024 19:43:36 GMT

I dont think there should an issue with it

Re: Need to define the struct and array of struct field colum in the delta live table(dlt) in databr

Data_Engineer3 — Tue, 23 Jan 2024 07:16:29 GMT

I have created DLT live tables pipeline, In Job UI, i can able to see only steps and if any failure happened it show only error at that stage.
But if i use any log using print, it doesn't show the logs in the console or any where. how can i see the logs associated with the pipelines. Is there any way to check the logs in DLT flow for development??How do I see the logs of the running DLT pipeline

Re: Need to define the struct and array of struct field colum in the delta live table(dlt) in databr

Lakshay — Tue, 23 Jan 2024 11:14:33 GMT

You should be able to access the logs from the right side panel under Update details. Please refer the screenshot below