cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
nolanlavender00
New Contributor
since ‎08-27-2021
‎06-26-2023

User Stats

  • 3 Posts
  • 0 Solutions
  • 0 Kudos given
  • 0 Kudos received

User Activity

Garbage Collection on AutoLoader

02-23-2023
Once a week, I get very long run times with AutoLoader. The spark job says it is done, but garbage collection keeps rising on the driver. I assume this is because of the backfill interval that I am using with FileNotification Type. I have this set to...

How to control garbage collection while using Autoloader File Notification?

02-10-2023
I am using Autoloader to load files from a directory. I have set up File Notification with the Event Subscription. I have a backfill interval set to 1 day and have not run the stream for a week. There should only be about ~100 new files to pick up an...

How to stop a Streaming Job based on time of the week

08-20-2021
I have an always-on job cluster triggering Spark Streaming jobs. I would like to stop this streaming job once a week to run table maintenance. I was looking to leverage the foreachBatch function to check a condition and stop the job accordingly.