apply a function across multiple smaller dataframes created from one big dataframe in scala

Sandesh87 — Tue, 13 Jun 2023 18:31:15 GMT

The dataframe 'big_df' looks like the below

| id| index| timestamp|

|:---- |:------:| -----:|

| abc| 1| 11:00:00|

| abc| 1| 11:00:10|

| abc| 1| 11:00:20|

| abc| 1| 11:00:30|

| abc| 1| 11:00:40|

| abc| 1| 11:00:50|

| abc| 2| 11:01:00|

| abc| 2| 11:01:10|

| abc| 2| 11:01:20|

| def| 1| 23:00:00|

| def| 1| 23:01:00|

| xyz| 1| 15:00:00|

| xyz| 1| 15:01:00|

| xyz| 1| 15:02:00|

| xyz| 1| 15:03:00|

| xyz| 1| 15:04:00|

| xyz| 1| 15:05:00|

| xyz| 2| 15:06:00|

| xyz| 2| 15:07:00|

| xyz| 3| 15:10:00|

There is a function 'fun1' which takes a dataframe as input.

Each unique combination of columns 'id' and 'index' in big_df is a small dataframe that needs to be passed to the function fun1.

How can this function be applied across multiple of the small dataframes in parallel?

Can it be achieved using the foreachpartition and if so how?

Re: apply a function across multiple smaller dataframes created from one big dataframe in scala

Anonymous — Fri, 16 Jun 2023 06:10:06 GMT

Hi @Sandesh Puligundla

Great to meet you, and thanks for your question!

Let's see if your peers in the community have an answer to your question. Thanks.