How does Catalyst optimizer improves the performances, what is its role?
Catalyst optimizer converts unresolved logical plan into executable physical plan, deep dive is available here
It would be great if Databricks could support WAFs in the near future. Our use case is to prevent data downloads and egress except from a few allowed IPs (e.g. company VPN). We do still want to allow workspace login from all IPs however.
Hi AllI am getting this error for some jobs. Can you please let me know what could be the reasonRun result unavailable: job failed with an error message -Run result unavailable: job failed with error messageUnexpected failure while waiting for the cl...
This is an issue on the cloud level so try to put retries in the job as it happens not for all cluster start , it may fails once but will start after retry,Also, raise a databricks ticket , they will provide permanent solution
I can see connectivity issues in my driver logs. How to solve this issue?
Can you give us some more error please, I hope you will get more error in logs, whether it is a connection issue because of JDbc URL or host name or password,something like this
Environment Tab in the Spark UI
In spark UI - > Environment Tab
I would like to know how to get the total size of my Delta table
The following Kb will show a step by step example on how to get the size of a Delta table https://kb.databricks.com/sql/find-size-of-table.html
Im getting the following error message when trying to use display()Internal error, sorry. Attach your notebook to a different cluster or restart the current cluster.com.databricks.rpc.RPCResponseTooLarge: rpc response (of 20978566 bytes) exceeds limi...
It seems like the error is coming from the 20MB output limit. For more information please check this https://docs.databricks.com/jobs.html#output-size-limits
I have a case where I don't know the user Id of the user but I have emails of the user whose Id have to delete , is it possible to provide email id in SCIM Api to delete the user Idhttps://docs.microsoft.com/en-us/azure/databricks/dev-tools/api/late...
High level:Check this out for a detailed comparison - https://databricks.com/spark/comparing-databricks-to-apache-spark
There are two types of auto scaling in Databricks: Standard and Optimized. In both scenarios when tasks are submitted the cluster will begin scaling to execute as many of them in parallel immediately.Scaling down is different. In optimized autoscalin...
Yes, you can alerts - Email alerts sent in case of job failure, success, or timeout. You can set alerts up for job start, job success, and job failure (including skipped jobs), providing multiple comma-separated email addresses for each alert type. Y...
Are there recommendations and/or examples of leveraging AWS and Azure with Databricks? If so, is there any best practices to follow? Want to ensure we avoid expensive data transfer across clouds
I imported one workspace into another and noticed there were several instances of RESOURCE_DOES_NOT_EXIST errors because of the folder structure of the workspace (despite importing the workspace as well), see example below:Get: https://dbc-9d482d3a-f...
Hi Brinda, it's daily. https://docs.databricks.com/administration-guide/account-settings/billable-usage-delivery.html#high-level-flow
| User | Count |
|---|---|
| 1601 | |
| 736 | |
| 343 | |
| 284 | |
| 247 |
