Databricks New Runtime Version is Available Now PySpark memory profiling- Memory profiling is now enabled for PySpark user-defined functions. This pr...

Data Engineering

Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.

Databricks New Runtime Version is Available Now

PySpark memory profiling- Memory profiling is now enabled for PySpark user-defined functions. This provides information on memory increment, memory usage, and number of occurrences for each line of code in a user-defined function.

Dynamic pruning for DELETE and UPDATE

DELETE

and

UPDATE

now use dynamic file and partition pruning in instances where it improves performance. For example, dynamic pruning is enabled when a smaller source table is used to update or delete rows in a larger table.

Bug fixes

Fixed an issue with JSON parsing in Auto Loader where all columns were left as strings. Previously, cloudFiles.inferColumnTypes was not set or set to false and the JSON contained nested objects.

AviralBhardwaj