@MarianoRanu SQL Collation support is now in private preview. If you have interest please reach out the Databricks team responsible for your company's account.
More details about release types: https://docs.databricks.com/en/release-notes/release-ty...
@ranged_coop In addition to my previous message, checkpointing is not a Databricks behavior as you said, checkpointing is part of open source Spark structured streaming.
@dmytro, Autoscaling is managed by Databricks and it's logic is mostly automatic. But If you're planning on structured streaming for production I suggest you to go for a fixed amount of workers and limiting your streaming query input rate or create a...
@dmytro yes, it's possible to monitor the consumer lag through the streaming query metrics. Every cluster that runs a spark structured streaming query will log the metrics for each streaming batch in the driver logs and Spark UI. More details at Moni...
