How to implement pagination for delta table without reading entire table

BeenA_33 — Fri, 08 Mar 2024 15:36:41 GMT

Re: How to implement pagination for delta table without reading entire table

wise_owl — Mon, 16 Sep 2024 11:08:42 GMT

You can leverage this code base. It works as expected using "next_page_token" parameter-

Don't forget to mark this solution as correct if this helped you 🙃

import requests token = 'your token' url = 'your URL' params = {'expand_tasks': 'true'} header = {'Authorization': f'Bearer {token}'} while True: response = requests.get(url, headers=header, params=params) response_data = response.json() jobs = response_data.get("jobs", []) for job in jobs: settings = job.get('settings') task = settings.get('tasks') if task and task[0].get('existing_cluster_id'): job_name = settings.get('name') job_creator = job.get('creator_user_name') print(f'job creator name= {job_creator} & job name= {job_name}') else: print(f"{settings.get('name')} not running on ACL") next_page_token = response_data.get('next_page_token') if not next_page_token: break params['page_token'] = next_page_token

topic How to implement pagination for delta table without reading entire table in Get Started Discussions

How to implement pagination for delta table without reading entire table

Re: How to implement pagination for delta table without reading entire table