Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
08-08-2025 12:50 AM - edited 08-08-2025 12:54 AM
Oh, I didn't notice that you don't have checkpoint. So I guess that's the reason of your issue. You must specify the checkpointLocation option before you run a streaming query. As I replied in different topic, autoloader under the hood is based on spark structured streaming and I Istrongly recommend that you read the overview of spark structured streaming. It should clarify a lot of concepts for you like how streaming query, checkpoints and many more.
PS. To find driver logs go to Compute -> click on your cluster -> Driver logs