cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
cancel
Showing results for 
Search instead for 
Did you mean: 

Populating a array of date tuples Scala

Techmate
New Contributor

Hi Friends

i am trying to pass a list of date ranges needs to be in the below format.

val predicates =

Array(

“2021-05-16” → “2021-05-17”,

“2021-05-18” → “2021-05-19”,

“2021-05-20” → “2021-05-21”)

I am then using map to create a range of conditions that will be passed to the jdbc method.

val predicates =

Array(

“2021-05-16” → “2021-05-17”,

“2021-05-18” → “2021-05-19”,

“2021-05-20” → “2021-05-21”)…map { case (start, end) =>

s"cast(NEW_DT as date) >= date ‘$start’ AND cast(NEW_DT as date) <= date ‘$end’"

The process will need to run daily and i need to dynamically populate these values as i cannot use the hard coded way.

Need help in how i can return these values from a method with incrementing start_date and end_date tuples that can generate like above.I had a mere idea like below but as i am new to scala not able to figure out. Please help

def predicateRange( start_date: String,

end_date: String): Array[(String,String)]= {

// iterate over the date values and add + 1 to both start and end and return the tuple

}

1 REPLY 1

-werners-
Esteemed Contributor III

So basically this can be done by generating 2 lists which are then zipped.

One list contains the first dates of the tuples, so these are in your case 2 days apart.

The other list is the 2nd dates of the tuples, also 2 days apart.

Now we need a function which creates these lists.

Mind that I found this function on the internet somewhere (forgot where) but it is elegant and works great!

import java.time.LocalDate

//create an infinite stream of dates starting at `fromDate`.

//streams are lazy evaluated so they are only filled as requested

def dates(fromDate: LocalDate): Stream[LocalDate] = {

 fromDate #:: dates(fromDate plusDays 2 ) //add 2 days

}

val x = dates(LocalDate.now()).take(5).toList //start from today

val y = dates(LocalDate.now().plusDays(1)).take(5).toList //start from tomorrow

val list = x zip y

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.