@skarpeck does your input df contain any filters? The empty codes variable could be due to empty microbatches maybe. Please check the numInputRows from your query's Stream Monitoring Metrics. I recommend you to check if there are input rows for the b...
@MarianoRanu SQL Collation support is now in private preview. If you have interest please reach out the Databricks team responsible for your company's account.
More details about release types: https://docs.databricks.com/en/release-notes/release-ty...
@ranged_coop In addition to my previous message, checkpointing is not a Databricks behavior as you said, checkpointing is part of open source Spark structured streaming.
@dmytro, Autoscaling is managed by Databricks and it's logic is mostly automatic. But If you're planning on structured streaming for production I suggest you to go for a fixed amount of workers and limiting your streaming query input rate or create a...
