cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Populating a array of date tuples Scala

Techmate
New Contributor

Hi Friends

i am trying to pass a list of date ranges needs to be in the below format.

val predicates =

Array(

“2021-05-16” → “2021-05-17”,

“2021-05-18” → “2021-05-19”,

“2021-05-20” → “2021-05-21”)

I am then using map to create a range of conditions that will be passed to the jdbc method.

val predicates =

Array(

“2021-05-16” → “2021-05-17”,

“2021-05-18” → “2021-05-19”,

“2021-05-20” → “2021-05-21”)…map { case (start, end) =>

s"cast(NEW_DT as date) >= date ‘$start’ AND cast(NEW_DT as date) <= date ‘$end’"

The process will need to run daily and i need to dynamically populate these values as i cannot use the hard coded way.

Need help in how i can return these values from a method with incrementing start_date and end_date tuples that can generate like above.I had a mere idea like below but as i am new to scala not able to figure out. Please help

def predicateRange( start_date: String,

end_date: String): Array[(String,String)]= {

// iterate over the date values and add + 1 to both start and end and return the tuple

}

1 REPLY 1

-werners-
Esteemed Contributor III

So basically this can be done by generating 2 lists which are then zipped.

One list contains the first dates of the tuples, so these are in your case 2 days apart.

The other list is the 2nd dates of the tuples, also 2 days apart.

Now we need a function which creates these lists.

Mind that I found this function on the internet somewhere (forgot where) but it is elegant and works great!

import java.time.LocalDate

//create an infinite stream of dates starting at `fromDate`.

//streams are lazy evaluated so they are only filled as requested

def dates(fromDate: LocalDate): Stream[LocalDate] = {

 fromDate #:: dates(fromDate plusDays 2 ) //add 2 days

}

val x = dates(LocalDate.now()).take(5).toList //start from today

val y = dates(LocalDate.now().plusDays(1)).take(5).toList //start from tomorrow

val list = x zip y

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group