Databricks Runtime 10.5 (Beta)

👉 Auto Loader: new SQL function CLOUD_FILES_STATE

You can use the new CLOUD_FILES_STATE function to query the internal state of an Auto Loader stream.

👉 Delta Lake: new maxRecordsPerFile option for maximum records written to a single file

When you use the DataFrame APIs to write to a Delta table, you can use the

"maxRecordsPerFile" option to specify the maximum number of records to write out to a single file. Setting a value of zero or a negative value represents no limit.

👉 Deprecation of Koalas

👉 Unity Catalog: SQL LIST output removes "is_directory" column and adds trailing / for directory paths...

👉 more info https://docs.databricks.com/release-notes/runtime/10.5.html