- 11865 Views
- 1 replies
- 0 kudos
Resolved! Why do I always see "Executor heartbeat timed out" messages in the Spark Driver logs
Often, I see "Executor heartbeat timed out" messages in the Spark driver logs. Sometimes job fails with this error. Will increasing "spark.executor.heartbeatInterval" help to mitigate the issue ?
- 11865 Views
- 1 replies
- 0 kudos
- 0 kudos
This is a common misconception that increasing "spark.executor.heartbeatInterval" will help to mitigate or resolve the heartbeat issues. In fact, increasing the spark.executor.heartbeatInterval will increase the chance of the error and worse the situ...
- 0 kudos
- 3885 Views
- 3 replies
- 0 kudos
How to check my streaming job's metrics?
I would like to know if there is a way to keep track of my running streaming job.
- 3885 Views
- 3 replies
- 0 kudos
- 0 kudos
Streaming metrics are available/exposed mainly through 3 ways:Streaming UI, which is available from Spark 3/DBR 7Streaming listener/Observable metrics APISpark driver logs. Search for the string "Streaming query made progress". The metrics are logged...
- 0 kudos
- 2426 Views
- 1 replies
- 0 kudos
- 2426 Views
- 1 replies
- 0 kudos
- 0 kudos
yes. it is actually quite fun to build Looker dashboards on top of delta lake. here is a Databricks on Looker tutorial that I created when Databricks on GCP was released.
- 0 kudos
- 2312 Views
- 1 replies
- 0 kudos
how do I change my workspace account owner?
If I need to change my account owner for my workspace, how would my team go about doing this?
- 2312 Views
- 1 replies
- 0 kudos
- 0 kudos
You need to create a ticket to data bricks support to change that
- 0 kudos
- 1173 Views
- 1 replies
- 0 kudos
Koalas or Pyspark
Should I use PySpark’s DataFrame API or Koalas, Which one is recommended , is there any performance impact if i use koalas or little slower than pyspark API
- 1173 Views
- 1 replies
- 0 kudos
- 0 kudos
If you are already familiar with pandas and want to leverage Spark for big data, we recommend using Koalas. If you are learning Spark from ground up, we recommend you start with PySpark’s API.
- 0 kudos
- 1405 Views
- 1 replies
- 0 kudos
Photon usage
How do I know how much of a query/job used Photon?
- 1405 Views
- 1 replies
- 0 kudos
- 0 kudos
If you are using Photon on Databricks SQLClick the Query History icon on the sidebar.Click the line containing the query you’d like to analyze.On the Query Details pop-up, click Execution Details.Look at the Task Time in Photon metric at the bottom.
- 0 kudos
- 3518 Views
- 1 replies
- 0 kudos
Malformed Request Error Message
I received the following error when launching a workspace: MALFORMED_REQUEST: Failed storage configuration validation checks: PUT, LIST, DELETE. How do I fix this?
- 3518 Views
- 1 replies
- 0 kudos
- 0 kudos
Check the S3 bucket policy and region. It looks like storage config validation is failing
- 0 kudos
- 1791 Views
- 1 replies
- 0 kudos
Failed E2 Workspace Error Message
My workspace ended up in a FAILED workspace state with one of the following messages:INVALID_STATE: The maximum number of VPCs has been reached.INVALID_STATE: The maximum number of VPC endpoints has been reached.INVALID_STATE: The maximum number of a...
- 1791 Views
- 1 replies
- 0 kudos
- 0 kudos
The maximum number of xxx has been reached indicates that you have hit the soft limits for some of the AWS resources in that region. These are mostly soft limits and you could file a request to AWS support team to increase this
- 0 kudos
- 1912 Views
- 1 replies
- 0 kudos
E2 Workspace DNS Unreachable
My E2 workspace is in a RUNNING state, but the DNS is unreachable.
- 1912 Views
- 1 replies
- 0 kudos
- 0 kudos
Try deleting the RUNNING workspace, wait for 5-10 minutes, and recreate the same workspace. If that doesn't solve the problem, file a support ticket
- 0 kudos
- 1773 Views
- 1 replies
- 0 kudos
E2 workspace - Error Message Malformed Request : Invalid xxx in the HTTP request body
I received one of the following errors: MALFORMED_REQUEST: Invalid xxx in the HTTP request body or MALFORMED_REQUEST: Invalid xxx in body, where xxx is credentials, storage configurations, networks, etc.
- 1773 Views
- 1 replies
- 0 kudos
- 0 kudos
It denotes that the input payload is not what is expected the api-endpoint for the e2 accounts api. Possible causes include typo in variable values or json formatting issues ( not providing quotes etc )
- 0 kudos
- 7953 Views
- 1 replies
- 1 kudos
- 7953 Views
- 1 replies
- 1 kudos
- 1 kudos
To access these driver log files from the UI, you could go to the Driver Logs tab on the cluster details page. You can also configure a log delivery location for the cluster. Both worker and cluster logs are delivered to the location you specify.
- 1 kudos
- 2117 Views
- 1 replies
- 0 kudos
- 2117 Views
- 1 replies
- 0 kudos
- 0 kudos
When you remove a user from Databricks, a special backup folder is created in the workspace. More details at https://kb.databricks.com/notebooks/get-notebooks-deleted-user.html
- 0 kudos
- 1543 Views
- 1 replies
- 0 kudos
- 1543 Views
- 1 replies
- 0 kudos
- 0 kudos
If you have some of the features like "secure cluster connectivity", "multi-workspace accounts" , chances are that the account is E2. Would recommend to check with your Databricks accounts team.
- 0 kudos
- 1854 Views
- 1 replies
- 0 kudos
Saving charts in S3
Can you save the charts that you create in Databricks notebooks as an image file to s3? For example I'm viewing the dataframe using display(df) and then using the plot button to actually create the graph
- 1854 Views
- 1 replies
- 0 kudos
- 0 kudos
Would rendering visualizations via plotly and saving them to s3 work ?
- 0 kudos
- 3073 Views
- 1 replies
- 1 kudos
SQL Formatting
Is there a way for individual users to adjust the SQL formatting defaults?
- 3073 Views
- 1 replies
- 1 kudos
- 1 kudos
Do you want a different format than Databrick provide the formatting in Sql cell?
- 1 kudos
| User | Count |
|---|---|
| 1644 | |
| 793 | |
| 553 | |
| 349 | |
| 287 |
