topic Re: Dynamically specify pivot column in SQL in Data Engineering

Dynamically specify pivot column in SQL

erigaud — Thu, 14 Dec 2023 07:50:24 GMT

Hello everyone !

I am looking for a way to dynamically specify pivot columns in a SQL query, so it can be used in a view. However we don't want to hard code the values that need to become columns, and would rather extract it from another table.

I've seen other posts on internet asking the same question but without any luck 😕

Here's what we have currently :

SELECT * FROM table PIVOT (sum(my_value)
FOR mapping in ('col1', 'col2', ...)

Instead we want to use a table table_mapping containing all the values that should go in the list, is there any way to do that in sql ?

Thank you !

Re: Dynamically specify pivot column in SQL

tmeyer — Tue, 03 Dec 2024 14:20:39 GMT

I'm facing the same issue as you do, and apart from the Pyspark way (specifiying a `.groupBy(cols,...).pivot(col)`) and putting it in a view to query from SQL, I didn't find a convenient Spark SQL way of doing it.

Is there any plan to add a feature to get dynamic columns in SQL for Pivot on the Databricks side?

Re: Dynamically specify pivot column in SQL

Wikram — Fri, 31 Jan 2025 17:46:14 GMT

Did u find any answer? Can u share your thoughts on meeting this usecase?

Re: Dynamically specify pivot column in SQL

lprevost — Mon, 09 Jun 2025 12:44:52 GMT

I agree - not clear how to do this? I'm thinking of using the pandas api.