cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Databricks New Runtime Version is Available Now  PySpark memory profiling- Memory profiling is now enabled for PySpark user-defined functions. This pr...

Aviral-Bhardwaj
Esteemed Contributor III

Databricks New Runtime Version is Available Now

image 

PySpark memory profiling- Memory profiling is now enabled for PySpark user-defined functions. This provides information on memory increment, memory usage, and number of occurrences for each line of code in a user-defined function.

Dynamic pruning for DELETE and UPDATE

DELETE

 and 

UPDATE

 now use dynamic file and partition pruning in instances where it improves performance. For example, dynamic pruning is enabled when a smaller source table is used to update or delete rows in a larger table.

Bug fixes

Fixed an issue with JSON parsing in Auto Loader where all columns were left as strings. Previously, cloudFiles.inferColumnTypes was not set or set to false and the JSON contained nested objects.

0 REPLIES 0
Join 100K+ Data Experts: Register Now & Grow with Us!

Excited to expand your horizons with us? Click here to Register and begin your journey to success!

Already a member? Login and join your local regional user group! If there isn’t one near you, fill out this form and we’ll create one for you to join!