An application receives messages from event hub. Below is a message received from event hub and loaded into a dataframe with one column
name,gender,id
sam,m,001
-----
time,x,y,z,long,lat
160,22,45,51,83,56
230,82,95,48,18,26
-----
event,a,b,c
034,1,5,6
073,4,2,8
Each message may contain 3 datasets separated by the five dashes -----
dataset1:
name,gender,id
sam,m,001
name,gender,id is header information in dataset1
dataset2:
time,x,y,z,long,lat
160,22,45,51,83,56
230,82,95,48,18,26
time,x,y,z,long,lat is header information in dataset2
dataset3:
event,a,b,c
034,1,5,6
073,4,2,8
event,a,b,c is the header information in dataset3
The application is a spark streaming one and batches multiple such messages into one dataframe. For example a dataframe with one column that loads three messages from event hub could look like below
name,gender,id
sam,m,001
-----
time,x,y,z,long,lat
160,22,45,51,83,56
230,82,95,48,18,26
-----
event,a,b,c
034,1,5,6
073,4,2,8
name,gender,id
janet,f,002
-----
time,x,y,z,long,lat
839,22,08,81,91,23
110,42,68,31,74,45
name,gender,id
ross,m,003
-----
time,x,y,z,long,lat
209,33,10,11,61,47
230,82,95,48,18,26
246,91,82,92,28,98
230,03,62,56,02,42
-----
event,a,b,c
034,4,1,0
092,9,8,3
dataset with header information event,a,b,c may or may not be present in the message as can be seen in the message with name 'janet' in the above.
Objective is to combine datasets 1 and 2 related to the specific message. dataset3 is excluded. The result should look like:
name gender id time x y z long lat
sam m 001 160 22 45 51 83 56
sam m 001 230 82 95 48 18 26
janet f 002 839 22 08 81 91 23
janet f 002 110 42 68 31 74 45
ross m 003 209 33 10 11 61 47
ross m 003 230 82 95 48 18 26
ross m 003 246 91 82 92 28 98
ross m 003 230 03 62 56 02 42
How to achieve this using scala?
