JUPin
Databricks Partner

Thanks for responding,

I've investigated your suggestions, here are my findings:

  1. Check the max_results parameter: Ensure that you’re not inadvertently limiting the number of results returned. The default value is 1000, but you can adjust it as needed.  -- I've adjusted this over several runs.  The results get very wonky when I have a hard set value, for example, if I put set "max_results=1000", I get an error message stating the maximum value can be only 250.  If I set it to 100 (for example), sometimes the "display()" statements stop working altogether.  I have to detach and reattach the compute cluster for it start working again.  If I set it from 10 to 25, the results consistently retrieve, 35 rows.

  2. Inspect the filter criteria: If you’re using any filters (such as level='INFO' or timestamp > 'TIMESTAMP'), review them to make sure they’re not unintentionally restricting the results. -- Yes I've tried the filters, this doesn't seem to make a difference.  As a suggestion, I would strongly encourage a filter on the "update_id".

  3. Pagination: The API response includes pagination tokens (next_page_token and prev_page_token). Make sure you’re handling these tokens correctly to retrieve all available events. If you’re not using them, you might be getting only the first page of results. -- Yes, I use "next_page_token" in my subsequent API calls.  Depending on how I set my "max_results", for example "max_results=25", I get the original data pull, then I use the "next_page_token" to get the next set, which is 10.  The second set doesn't have a "next_page_token"

  4. Rate Limiting: Check if there’s any rate limiting or throttling applied to your API requests. Some APIs limit the number of requests per minute or hour. -- I don't receive any rate limiting error.  The API continues to call until it receives no response, I can even do it manually, so I don't believe this is an issue

  5. Error Handling: Inspect the response for any error messages or status codes. It’s possible that an error is occurring during the API call. -- I've checked all the error messages and status codes that return, I do not receive any errors.

Currently, I'm trying to setup a very simple example for the API call issue and the SDK to upload.