The documentation has changed, and that 20MB log limit is now documented in https://docs.databricks.com/en/jobs/how-to/use-jars-in-workflows.html#output-size-limits-for-jar-jobs.
Soo.. You use a loop to go through metadata table and then retrieve and ingest files using JDBC?If so, then the concurrent runs won't be helpful. Concurrent runs means the number of how many runs of that job can be ran side by side. For you, this wou...
I spent some time configuring a setup similar to this. Unfortunately, there's no simple way to do this. There's only {{job.trigger.file_arrival.location}} parameter, but that is pretty much useless, since it points to the directory that we are watchi...
