Databricks Runtime 10.5 (Beta) �� Auto Loader: new SQL function CLOUD_FILES_STATE You can use the new CLOUD_FILES_STATE function to query the ...

[Hubert-Dudek]

Databricks Runtime 10.5 (Beta)

👉 Auto Loader: new SQL function CLOUD_FILES_STATE

You can use the new CLOUD_FILES_STATE function to query the internal state of an Auto Loader stream.

👉 Delta Lake: new maxRecordsPerFile option for maximum records written to a single file

When you use the DataFrame APIs to write to a Delta table, you can use the

"maxRecordsPerFile" option to specify the maximum number of records to write out to a single file. Setting a value of zero or a negative value represents no limit.

👉 Deprecation of Koalas

👉 Unity Catalog: SQL LIST output removes "is_directory" column and adds trailing / for directory paths...

👉 more info https://docs.databricks.com/release-notes/runtime/10.5.html

[SørenRavn]

anyone that got CLOUD_FILES_STATE to work on 10.5 Beta/Azure.

Syntax:

%sql

SELECT * FROM cloud_files_state('abfss://test@test.dfs.core.windows.net/test/checkpoint/');

i get this error:

com.databricks.backend.common.rpc.DatabricksExceptions$SQLExecutionException: org.apache.spark.sql.AnalysisException: could not resolve `cloud_files_state` to a table-valued function