cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

Need to define the struct and array of struct field colum in the delta live table(dlt) in databrick.

Data_Engineer3
Contributor III

I want to create the columns with datatype struct and array of struct datatype in the DLT live tables, will it be possible, if possible could you share the sample for the same.

Thanks.

3 ACCEPTED SOLUTIONS

Accepted Solutions

Lakshay
Esteemed Contributor

Hi @Data_Engineer3 , In DLT, we don't create empty tables to begin with but we create the tables by reading the source. For ex:-

CREATE STREAMING LIVE TABLE customers
AS SELECT * FROM cloud_files("/databricks-datasets/retail-org/customers/", "csv");

 If your source data has the struct and array columns, your DLT tables will also have these columns as a result.  

View solution in original post

 

Hi @Lakshay 

Thanks for reply,

Yes, my transformation query(i.e dlt view) will have the columns with type struct field like(Column_name) below

Column_O STRING,

Column_name STRUCT< vaccines
ARRAY<
STRUCT< vaccineId STRING, date STRING > >
satisfiedBy ARRAY< STRUCT< vaccineId STRING, orderId STRING, orderItemId STRING >>
serviceName STRING,
dueDate STRING,
completedAt STRING,
state ARRAY< STRUCT<id STRING, name STRING>>,
nextDueDate STRING
>,

Column_name_2   STRING,

.is it possible to create the column with above structure, basically it is combination of both structure and array of structure.

  Thanks

View solution in original post

Lakshay
Esteemed Contributor

You should be able to access the logs from the right side panel under Update details. Please refer the screenshot below

Screenshot 2024-01-23 at 4.42.44โ€ฏPM.png

View solution in original post

5 REPLIES 5

Lakshay
Esteemed Contributor

Hi @Data_Engineer3 , In DLT, we don't create empty tables to begin with but we create the tables by reading the source. For ex:-

CREATE STREAMING LIVE TABLE customers
AS SELECT * FROM cloud_files("/databricks-datasets/retail-org/customers/", "csv");

 If your source data has the struct and array columns, your DLT tables will also have these columns as a result.  

 

Hi @Lakshay 

Thanks for reply,

Yes, my transformation query(i.e dlt view) will have the columns with type struct field like(Column_name) below

Column_O STRING,

Column_name STRUCT< vaccines
ARRAY<
STRUCT< vaccineId STRING, date STRING > >
satisfiedBy ARRAY< STRUCT< vaccineId STRING, orderId STRING, orderItemId STRING >>
serviceName STRING,
dueDate STRING,
completedAt STRING,
state ARRAY< STRUCT<id STRING, name STRING>>,
nextDueDate STRING
>,

Column_name_2   STRING,

.is it possible to create the column with above structure, basically it is combination of both structure and array of structure.

  Thanks

Lakshay
Esteemed Contributor

I dont think there should an issue with it

Data_Engineer3
Contributor III

I have created DLT live tables pipeline, In Job UI, i can able to see only steps and if any failure happened it show only error at that stage.
But if i use any log using print, it doesn't show the logs in the console or any where. how can i see the logs associated with the pipelines. Is there any way to check the logs in DLT flow for development??How do I see the logs of the running DLT pipeline

Lakshay
Esteemed Contributor

You should be able to access the logs from the right side panel under Update details. Please refer the screenshot below

Screenshot 2024-01-23 at 4.42.44โ€ฏPM.png

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you wonโ€™t want to miss the chance to attend and share knowledge.

If there isnโ€™t a group near you, start one and help create a community that brings people together.

Request a New Group