Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
09-11-2022 08:39 PM
I am assuming the schema of all these files is same
If so how to process it depends what your comfortable with
The steps that come to mind are
- In the landing zone have a folder structure per client
- read all the parquet contract files into delta input_file_name() may be useful to know which file your processing
(Contracts are per client with a type and start end date)
- Create a column for client name
Perform aggregations
- how many different contract did the client have
---group by clientname and a count
- of which type
--group by clientname and distinct count on type
- when was the dateof the first contract
--group by clientname and min date
- and the last contract
--group by clientname and max date
- how long have we been working with him.
--difference between min and max