Can you run non-spark jobs on Databricks?

User16826992666
Databricks Employee
Databricks Employee

Is spark the only type of code that can run on a Databricks cluster?

User16826992666
Databricks Employee
Databricks Employee

You can also run straight python code on Databricks clusters, but it will not run in a distributed manner.

sajith_appukutt
Databricks Employee
Databricks Employee

Databricks has a Runtime for Machine Learning that comes with a lot of libraries/frameworks pre-installed. This allows you to run for example PyTorch / TensorFlow code without worrying about infrastructure setup, configuration and dependency management.

Databricks also supports single node clusters to run single-node machine learning workloads