cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Help
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Forum Posts

Start a conversation
weldermartins
by • Honored Contributor
  • 5172 Views
  • 2 replies
  • 1 kudos

Resolved! How to make spark-submit work on windows?

I have Jupyter Notebook installed on my machine working normally. I tested running a Spark application by running the spark-submit command and it returns the message that the file was not found. What do you need to do to make it work?Below is a file ...

image
Data Engineering
Reply
  • 5172 Views
  • 2 replies
  • 1 kudos
View Replies
Latest Reply
Debayan
Databricks Employee
  • 1 kudos

Hi, yet this is not tested in my lab, but could you please check and confirm if this works: https://stackoverflow.com/questions/37861469/how-to-submit-spark-application-on-cmd

  • 1 kudos
1 More Replies
Ashish
by • New Contributor II
  • 7478 Views
  • 4 replies
  • 3 kudos

Resolved! Cost of individual jobs running on a shared Databricks cluster

Hi All,I am working on a requirement where I need to calculate the cost of each spark job individually on a shared Azure/AWS Databricks cluster. There can be multiple jobs running on the cluster parallelly.Cost needs to be calculated after job comple...

Data Engineering
Reply
  • 7478 Views
  • 4 replies
  • 3 kudos
View Replies
Latest Reply
alexott
Databricks Employee
  • 3 kudos

There is a built-in functionality for getting the costs:AWS - https://docs.databricks.com/administration-guide/account-settings-e2/usage.htmlAzure - via built-in Cost Management + BillingThe main problem with that functionality is that the smallest g...

  • 3 kudos
3 More Replies
Andriy_Shevchen
by • New Contributor
  • 3031 Views
  • 2 replies
  • 3 kudos

Resolved! yarn.nodemanager.resource.memory-mb parameter update

I am currently working on determining proper cluster size for my Spark application and I have a question regarding Hadoop configuration parameter yarn.nodemanager.resource.memory-mb. From what I see, this parameter is responsible for setting the phys...

Data Engineering
Reply
  • 3031 Views
  • 2 replies
  • 3 kudos
View Replies
Latest Reply
jose_gonzalez
Databricks Employee
  • 3 kudos

Hi @Andriy Shevchenko​ ,Databricks does not use Yarn. I recommend you to try to use Databricks community edition link to get familiar and explore. You can check Ganglia UI to see how is the cluster utilization, memory, cpu, IO, etc

  • 3 kudos
1 More Replies
Labels