- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-20-2026 12:04 PM
Hi @antgei ,
Thanks for sharing your experience and the workaround. You raise a valid point -- the platform should ensure task files are fully synced before attempting execution, regardless of API rate limiting on the backend. When a job has hundreds of SQL file tasks referencing files in a Git folder (accessed via the /Workspace/ path), the scheduler needs to fetch and validate each one, and under heavy load it can hit internal rate limits that cause transient "Failed to fetch SQL file" errors even though the files exist.
Your workaround of wrapping the SQL-tasks-job inside a parent job_task with retries is a reasonable approach for now. A few other things worth trying if you have not already:
- Consolidate SQL files where possible to reduce the total number of file-fetch operations per job run
- Stagger task execution using task dependencies rather than launching all SQL tasks in parallel, which can reduce the burst of file-fetch API calls
- If using Git folders, consider copying the SQL files to a regular Workspace directory (non-Git-backed) before the job runs, since Git-backed workspace paths have additional sync overhead
That said, I agree this behavior warrants a product improvement. I will flag this internally to ensure the engineering team is aware of the pattern. If you have not already, I would also recommend opening a support ticket referencing this thread so it gets tracked formally.