PriyaAnanthram
Contributor III

I am assuming the schema of all these files is same

If so how to process it depends what your comfortable with

The steps that come to mind are

  • In the landing zone have a folder structure per client
  • read all the parquet contract files into delta input_file_name() may be useful to know which file your processing

(Contracts are per client with a type and start end date)

  • Create a column for client name

Perform aggregations

  • how many different contract did the client have

---group by clientname and a count

  • of which type

--group by clientname and distinct count on type

  • when was the dateof the first contract

--group by clientname and min date

  • and the last contract

--group by clientname and max date

  • how long have we been working with him.

--difference between min and max