Not sure if this has come up before, but I'm wondering if Databricks has any kind of functionality to "watch" an API call for changes?
E.g. Currently I have a frequently running job that pulls data via an API call and overwrites the old data. This seems pretty inefficient since the data from the API call may or may not change between each overwrite. In fact, the data from the API call changes in an "ad-hoc" basis so it would be nice to pull data from the API *only* when there are data changes.
