cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Pyspark Convert Struct Type to Map Type

Anbazhagananbut
New Contributor II

Hello Sir,

Could you please advise the below scenario in pyspark 2.4.3 in data-bricks

to load the data into the delta table.

I want to load the dataframe with this column "data" into the table as Map

type in the data-bricks spark delta table.could you please advise on this

scenario.how to convert struct type into map type

i have a column named "data" which is struct type. pls find the sample data

format:

data:struct

field1:long

field2:string

field3:array

field4:long

field5:struct

code:string

name:string

field6:struct

lat:double

lon:double

field7:long

field8:string

field9:boolean

field10:long

field11:string

field12:string

field13:long

field14:boolean

field15:array

element:string

id:string

field16:array

element:string

Thanks

2 REPLIES 2

shyam_9
Databricks Employee
Databricks Employee

Hi @Anbazhagan anbutech17,

you can do this by explode(), please follow the example in the below notebook,

https://docs.azuredatabricks.net/_static/notebooks/transform-complex-data-types-python.html

sherryellis
New Contributor II

you can do it by making an api request - convert png to ico paint/api/2.0/clusters/permanent-delete

i dont see an option to delete or edit an automated cluster from UI.

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group