This behavior is caused by the way the Databricks CLI currently handles recursive globbing for the include section in databricks.yml files. You are not misunderstanding; this is a limitation (and partially a bug) in how the CLI resolves glob patterns for included YAML files rather than a mistake in your configuration.
Explanation
According to the Databricks Asset Bundle configuration documentation, entries in include are processed using relative path globs that behave similarly to .gitignore patterns. However, the CLI does not support fully recursive multi-level expansion beyond one level of double-star (**) recursion in some contexts. Specifically:
-
resources/**/*.yml expands only one directory deep under resources, meaning files like resources/clusters/cluster*.yml are correctly found.
-
When deeper folders exist (for example, resources/unity-catalog/schemas/schema1.yml), the ** pattern stops matching because the CLI performs only one recursive depth expansion during include resolution, not fully recursive searching.
-
Conversely, when you change the pattern to resources/**/**/*.yml, the traversal starts deeper, catching nested schemas but skipping the shallower directory, since it shifts the base recursion depth.
This behavior affects the include array in databricks.yml but not the sync.include or sync.paths mappings, which use standardized .gitignore-style globbing.
Confirmation of the Bug
A related issue was raised in Databricks CLI GitHub issue #1755, where similar path resolution and glob-root expansion inconsistencies were reported. The Databricks team acknowledged the issue and implemented a patch (#1756 – Expand library globs relative to the sync root) to improve consistent glob expansion, but as of version 0.227.x–0.228.x, this fix primarily affected the sync and resource path resolution, not include.
Hence your observation—having to specify multiple patterns (e.g., both resources/**/*.yml and resources/**/**/*.yml)—is consistent with this incomplete glob-expansion behavior.
Workarounds
Until Databricks extends consistent glob handling to nested includes, you can:
-
Combine multiple patterns in your include mapping:
include:
- resources/**/*.yml
- resources/**/**/*.yml
-
Alternatively, flatten the resource structure or consolidate schema definitions where possible.
-
Ensure your CLI is updated beyond v0.228.x once a fix for recursive include resolution is officially released.
Summary
You’re not misunderstanding the syntax; it’s a known, unresolved bug in the Databricks CLI’s recursive include implementation. As of CLI versions through 0.228.x, the recursive ** matches inconsistently depending on directory depth within the include block.
