Databricks Community

Hi @Master_DataBric , Yes its possibleHere is the doc link : - https://docs.databricks.com/aws/en/dlt/expectations?language=Python- https://docs.databricks.com/aws/en/dlt/expectations?language=SQL

Databricks is highly optimized for Delta, which leverages columnar storage, indexing, and caching for better performance.Instead of directly processing CSV files, convert them to Delta first, then perform aggregations and joins, see if this helps

Hi @397973, Combining multiple notebooks into a single notebook isn't an out-of-the-box feature, but will try to combine %run commands ando output them to see if it works, sort of like: %run "/path/to/notebook1"%run "/path/to/notebook2"

When you create a Databricks Lakehouse Monitoring monitor with an Inference profile, the system automatically generates two metric tables: a profile metrics table and a drift metrics table. Here's how this process works: Background Processing When yo...

@ShivangiB You're correct in your understanding. When you change a clustering key using ALTER TABLE followed by OPTIMIZE, it doesn't automatically recluster existing data. Let me explain why this happens and what options you have.In Delta Lake (which...

Hello @HarryRichard08! It looks like this post duplicates the one you recently posted. A response has already been provided to the Original post. I recommend continuing the discussion in that thread to keep the conversation focused and organized.

It turns out, the problem is the documentation. It says that the permission parameter (that's supplied in) is an array of strings. It really just expects a string, either UNKNOWN, USER, or ADMIN. It would be great if the team could fix the documentat...

Hi @gauravmahajan,Most of the cost / DBU used can be retrieved from System tables across your different workspaces in a databricks account. Details related to job compute types and it's associated cost can be fetched from the queries mentioned in the...

@SB93 The error message you are seeing indicates that the cluster failed to launch because the Spark driver was unresponsive, with possible causes being library conflicts, incorrect metastore configuration, or other configuration issues. Given that t...

@Suncat There hasn't been any major changes for than a year: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-version-support E.g. I don't believe we will see support for Spark 3.5 at all. At least, apparently it's support...

@n1399 The error "Validation failed for azure_attributes.spot_bid_max_price from pool, the value must be present" suggests that the spot bid max price is required, but it’s either missing or not correctly inherited from the compute policy when using ...