Hi @jorperort I know there is no direct task like bash_task for jobs that allows you to run bash scripts without using notebook cells %sh or Python's subprocess. Have you considered using Init scripts for your cluster while setting up the job?

With the init script, you can execute bash scripts to download or even run commands on a cluster. You can try it and tell me how it goes

Hello @jorperort , I did some research internally and have some tips/suggestions for you to consider:

Key Points from Analysis

• Databricks Asset Bundles & Job Initialization

  • Asset bundles allow automated cloning and setup of code and assets, and can be configured to initialize environments, install dependencies, and set up scripts using init scripts.
  • There is extensive support and documentation for using init scripts (cluster-scoped, volume-based, or workspace files) to run Bash scripts or install binaries at cluster startup. However, init scripts only run automatically during cluster startup, and you cannot trigger them ad hoc after the cluster is up.
• Job Task Execution Model

  • The standard Databricks job execution model is highly abstracted. User jobs (from Workflows, job API, or the UI) always execute within a managed containerized environment, via notebooks, JARs, Python scripts, dbt, or other supported tasks. There is no supported mechanism to attach a job or task that instructs the cluster runtime to simply run a Bash script “raw” on the OS at job runtime.
  • Even with single node clusters (where the compute node functions as both driver and worker), the available user-accessible task types are still confined to managed abstractions (notebooks, scripts, SQL, etc.). The compute runs Spark processes and “job” context; directly launching OS-level processes without these abstractions is not supported nor exposed in the APIs or asset bundle configuration.
• No Shell/SSH Access

  • Databricks does not provide shell access (e.g., SSH) to job compute nodes, especially on ephemeral job clusters (including single node types). Any mechanisms that do interact with the node OS (such as init scripts) are provisioned and managed by system services and are not interactive or re-entrant during job/task execution.
• Init Scripts as Only "Native" Mechanism

  • All the current documentation and internal references state that cluster/job initialization scripts (init scripts) are the supported way to execute arbitrary Bash commands or install/run binaries on compute nodes.
  • These scripts must be attached at cluster creation time and run during startup. There is no documented means to attach or execute them at arbitrary points during a job run.
• Alternative Paths Not Supported

  • There are no API endpoints, CLI commands, or asset bundle configurations that would allow you to submit a raw OS-level command or script to be executed outside of the managed abstractions noted above.
  • All code examples and available templates in the Databricks SDKs, as well as job/task APIs, ultimately execute within these supported frameworks and do not provide an escape hatch to the raw OS command environment at runtime.

Explicit Limitations and Security Considerations

• Allowing arbitrary “raw” execution of binaries or Bash scripts on the underlying compute node would break the Databricks managed security model and is deliberately not permitted, both to ensure workspace isolation and prevent customer or third-party code from affecting the platform or other users.
• The ephemeral and containerized nature of job clusters means that even if a script or binary were deposited into the container, there is no “native” way to ask the node’s system to execute it post-startup, apart from the managed task environments.

Table: Supported vs. Unsupported Mechanisms

Conclusion