cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Pivot in Databricks SQL

Gilg
Contributor II

Hi Team,

I have a table that has a key column (column name) and value column (value of the column name). These values are generated dynamically and wanted to pivot the table.

Question 1: Is there a way that we can do this without specifying all the columns in the expression_list? 

i.e. 

 

 

 

select id, date, col1, col2, col3, col4
from table1
pivot (
 max(value) as a 
 for key in ('col1', 'col2', 'col3', 'col4') <-- how to do this without specifying each column
)
group by all

 

 

 

 Question 2: When I tried specifying a column in the expression_list (same code above)

getting these results

Gilg_0-1695088239719.png

 

Wanted to collapse the result so that it will only be one row.

Cheers,

G

 

 

2 REPLIES 2

atrev66
New Contributor II

you have to group by just the id columns and put functions on the rest of the columns you want. See below for solution. Yeah, it's tedious.

select id, date, max(col1) as col1, max(col2) as col2, max(col3) as col3, max(col4) as col4
from table1
pivot (
 max(value) as a 
 for key in ('col1', 'col2', 'col3', 'col4') <-- how to do this without specifying each column
)
group by id, date

 

NSonam
New Contributor II

PySpark can help to list the available columns .Please find the demo snippets as below 

Image 1Image 1.

 

Image 2Image 2

 

Join Us as a Local Community Builder!

Passionate about hosting events and connecting people? Help us grow a vibrant local community—sign up today to get started!

Sign Up Now