Databricks New Runtime Version is Available Now PySpark memory profiling- Memory profiling is now enabled for PySpark user-defined functions. This pr...

Aviral-Bhardwaj — Sun, 18 Dec 2022 05:46:48 GMT

Databricks New Runtime Version is Available Now

PySpark memory profiling- Memory profiling is now enabled for PySpark user-defined functions. This provides information on memory increment, memory usage, and number of occurrences for each line of code in a user-defined function.

Dynamic pruning for DELETE and UPDATE

DELETE

and

UPDATE

now use dynamic file and partition pruning in instances where it improves performance. For example, dynamic pruning is enabled when a smaller source table is used to update or delete rows in a larger table.

Bug fixes

Fixed an issue with JSON parsing in Auto Loader where all columns were left as strings. Previously, cloudFiles.inferColumnTypes was not set or set to false and the JSON contained nested objects.

topic Databricks New Runtime Version is Available Now PySpark memory profiling- Memory profiling is now enabled for PySpark user-defined functions. This pr... in Data Engineering

Databricks New Runtime Version is Available Now PySpark memory profiling- Memory profiling is now enabled for PySpark user-defined functions. This pr...