- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
06-03-2025 02:11 PM - edited 06-03-2025 02:15 PM
Hi @sparklez
You're encountering this issue because the libraries field is not valid in the cluster configuration.
Libraries need to be specified at the job level, not the cluster level.
Option 1: Job-Level Libraries (Recommended)
Move the libraries section to your job configuration:
Option 2: All-Purpose Cluster with Libraries:
If you need an all-purpose cluster with libraries, create it separately:
Key Points:
1. Job clusters: Libraries are specified in the job definition
2. All-purpose clusters: Libraries are installed after cluster creation, typically through job definitions that reference the cluster
3. The libraries field is not supported in standalone cluster configurations in Databricks Asset Bundles
The job-level approach (Option 1) is generally preferred as it ensures libraries are installed
when the job runs and provides better isolation.