Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
Hello. Is there an equivalent of SQL stored procedure in Databricks? Please note that I need a procedure that allows DML statements and not only Select statement as a function provides.Thank you in advance
Weekly Release Notes RecapHere’s a quick recap of the latest release notes updates from the past one week.Databricks platform release notesFebruary 21 - 28, 2023Ray on Databricks (Public Preview)With Databricks Runtime 12.0 and above, you can create ...
Databricks New Runtime Version is Available Now PySpark memory profiling- Memory profiling is now enabled for PySpark user-defined functions. This provides information on memory increment, memory usage, and number of occurrences for each line of code...
Hello. During some example cases we were running, in order to identify how Databricks treats possible wrong actions we could make, we noticed that merge doesn't fail while inserting different data type values from the ones in the corresponding table....
If I create delta table the table is stored in parque format in DBFS location ? and please share how the parque files supports schema evolution if i do DML operation.As per my understanding : we read data from data lake first in data frame and try to...
delta lake is parquet on steroids. The actual data is stored in parquet files, but you get a bunch of extra functionalities (time traveling, ACID, optimized writes, MERGE etc).check this page for lots of info.Delta lake does support schema evolution...
