- 5858 Views
- 3 replies
- 4 kudos
Resolved! Incremental write
Hi All,I have a daily spark job that reads and joins 3-4 source tables and writes the df in a parquet format. This data frame consists of 100+ columns. As this job run daily, our deduplication logic identifies the latest record from each of source t...
- 5858 Views
- 3 replies
- 4 kudos
- 1679 Views
- 0 replies
- 0 kudos
Incremental updates to s3 csv files, autoloader, and delta lake updates
I'm using the Databricks autoloader to incrementally load a series of csv files on s3 which I update with an API. My tyipcal work process is to update only the latest year file each night. But, there are ocassions where previous years also get update...
- 1679 Views
- 0 replies
- 0 kudos
- 5420 Views
- 3 replies
- 0 kudos
Get last modified date or create date for azure blob container
Hi Everyone, I am trying to implement a way in Python to only read files that weren't loaded since the last run of my notebook. The way I am thinking of implementing this is to keep of the last time my notebook has finished in a database table. Nex...
- 5420 Views
- 3 replies
- 0 kudos
- 0 kudos
Hello! I just wanted to share my point of view on the topic of dating sites. I have been looking for a decent Asian catch-up site for a very long time, in addition to them I found https://hookupsearch.org/asian-hookup-sites/. We definitely recommend...
- 0 kudos