Databricks Community

The following Kb will show a step by step example on how to get the size of a Delta table https://kb.databricks.com/sql/find-size-of-table.html

It seems like the error is coming from the 20MB output limit. For more information please check this https://docs.databricks.com/jobs.html#output-size-limits

High level:Check this out for a detailed comparison - https://databricks.com/spark/comparing-databricks-to-apache-spark

There are two types of auto scaling in Databricks: Standard and Optimized. In both scenarios when tasks are submitted the cluster will begin scaling to execute as many of them in parallel immediately.Scaling down is different. In optimized autoscalin...

Yes, you can alerts - Email alerts sent in case of job failure, success, or timeout. You can set alerts up for job start, job success, and job failure (including skipped jobs), providing multiple comma-separated email addresses for each alert type. Y...

Hi Brinda, it's daily. https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html#high-level-flow

If you are talking about distributed training of a single XGBoost model, there is no built-in capability in SparkML. SparkML supports gradient boosted trees, but not XGBoost specifically. However, there are 3rd party packages, such as XGBoost4J that ...

With Spark, there are a few ways you can scale your model: TrainingHyperparameter tuningInferenceIf you're looking to train one model across multiple workers, you can leverage Horovod. It's an open source project designed to simplify distributed neur...

Pandas works for single machine computations, so any pandas code you write on Databricks will run on the driver of the cluster. Pyspark and Koalas are both distributed frameworks for when you have large datasets. You can use Pyspark and Koalas inte...