Databricks Runtime 10.5 (Beta) ๏ฟฝ๏ฟฝ Auto Loader: new SQL function CLOUD_FILES_STATE You can use the new CLOUD_FILES_STATE function to query the ...
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ04-22-2022 02:38 AM
Databricks Runtime 10.5 (Beta)
๐ Auto Loader: new SQL function CLOUD_FILES_STATE
You can use the new CLOUD_FILES_STATE function to query the internal state of an Auto Loader stream.
๐ Delta Lake: new maxRecordsPerFile option for maximum records written to a single file
When you use the DataFrame APIs to write to a Delta table, you can use the
"maxRecordsPerFile" option to specify the maximum number of records to write out to a single file. Setting a value of zero or a negative value represents no limit.
๐ Unity Catalog: SQL LIST output removes "is_directory" column and adds trailing / for directory paths...
๐ more info https://docs.databricks.com/release-notes/runtime/10.5.html
- Labels:
-
Beta
-
Databricks Runtime
-
Delta
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
โ04-29-2022 04:03 AM
anyone that got CLOUD_FILES_STATE to work on 10.5 Beta/Azure.
Syntax:
%sql
SELECT * FROM cloud_files_state('abfss://test@test.dfs.core.windows.net/test/checkpoint/');
i get this error:
com.databricks.backend.common.rpc.DatabricksExceptions$SQLExecutionException: org.apache.spark.sql.AnalysisException: could not resolve `cloud_files_state` to a table-valued function

