cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

User16826992666
by Valued Contributor
  • 2432 Views
  • 1 replies
  • 0 kudos

Resolved! How does cluster autoscaling work?

What determines when the cluster autoscaling activates to add and remove workers? Also, can it be adjusted?

  • 2432 Views
  • 1 replies
  • 0 kudos
Latest Reply
sajith_appukutt
Honored Contributor II
  • 0 kudos

> What determines when the cluster autoscaling activates to add and remove workersDuring scale-down, the service removes a worker only if it is idle and does not contain any shuffle data. This allows aggressive resizing without killing tasks or recom...

  • 0 kudos
Anonymous
by Not applicable
  • 6434 Views
  • 1 replies
  • 0 kudos

Resolved! Scheduling cluster start and stop time

I want to schedule cluster to start in the morning and shut down by evening. How can I achieve that?

  • 6434 Views
  • 1 replies
  • 0 kudos
Latest Reply
Anonymous
Not applicable
  • 0 kudos

You can call the REST API to schedule cluster starts and stops from a scheduler.See https://docs.databricks.com/dev-tools/api/latest/clusters.htmlPRO Tip: Use code generation tools within Postman to generate scripts in the language of your choice.

  • 0 kudos
User16826994223
by Honored Contributor III
  • 1259 Views
  • 1 replies
  • 0 kudos

Unable to start cluster Error :- Defunct Resource Detected

Hi AllI am getting this error for some jobs. Can you please let me know what could be the reasonRun result unavailable: job failed with an error message -Run result unavailable: job failed with error messageUnexpected failure while waiting for the cl...

  • 1259 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 0 kudos

This is an issue on the cloud level so try to put retries in the job as it happens not for all cluster start , it may fails once but will start after retry,Also, raise a databricks ticket , they will provide permanent solution

  • 0 kudos
jose_gonzalez
by Databricks Employee
  • 1080 Views
  • 1 replies
  • 0 kudos

How to solve Hive connectivity issues?

I can see connectivity issues in my driver logs. How to solve this issue?

  • 1080 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 0 kudos

Can you give us some more error please, I hope you will get more error in logs, whether it is a connection issue because of JDbc URL or host name or password,something like this

  • 0 kudos
Anonymous
by Not applicable
  • 1877 Views
  • 1 replies
  • 0 kudos
  • 1877 Views
  • 1 replies
  • 0 kudos
Latest Reply
Ryan_Chynoweth
Esteemed Contributor
  • 0 kudos

There are two types of auto scaling in Databricks: Standard and Optimized. In both scenarios when tasks are submitted the cluster will begin scaling to execute as many of them in parallel immediately.Scaling down is different. In optimized autoscalin...

  • 0 kudos
jose_gonzalez
by Databricks Employee
  • 1281 Views
  • 2 replies
  • 0 kudos

Cluster goes unresponsive after installing a library

Right after I install a library in my cluster, my cluster goes unresponsive and nothing runs. How to solve this issue?

  • 1281 Views
  • 2 replies
  • 0 kudos
Latest Reply
jose_gonzalez
Databricks Employee
  • 0 kudos

it is a standard cluster. It is happening for all libraries. is there a way to debug or show the errors messages if any?

  • 0 kudos
1 More Replies
User16783853032
by Databricks Employee
  • 1753 Views
  • 1 replies
  • 0 kudos

Databricks notebook command gets cancelled:Generally when cluster is having init scripts or lib issues while starting cluster. Exact error can be look...

Databricks notebook command gets cancelled:Generally when cluster is having init scripts or lib issues while starting cluster. Exact error can be looked into driver logs.

Screen Shot 2021-06-07 at 2.42.14 PM Screen Shot 2021-06-07 at 2.45.22 PM
  • 1753 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16826994223
Honored Contributor III
  • 0 kudos

Awsome Knowledge

  • 0 kudos
User16790091296
by Contributor II
  • 2682 Views
  • 1 replies
  • 0 kudos
  • 2682 Views
  • 1 replies
  • 0 kudos
Latest Reply
User16790091296
Contributor II
  • 0 kudos

Databricks starts to charge for DBUs once the virtual machine is up and the Spark context is initialized, which may include a portion of start up costs, but not all. Init scripts are loaded before the Spark context is initialized, which therefore wou...

  • 0 kudos
Van-DuyetLe
by New Contributor
  • 37989 Views
  • 5 replies
  • 3 kudos

What's the difference between Interactive Clusters and Job Cluster?

I am new to databricks. I would like to know what is the difference between Interactive Clusters and Job Cluster? There are no official document now.

  • 37989 Views
  • 5 replies
  • 3 kudos
Latest Reply
Forum_Admin
Contributor
  • 3 kudos

Sports news Football news International football news Football news Thai football news, Thai football Follow news, know sports news at Siamsportnews

  • 3 kudos
4 More Replies
Labels