- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
a week ago - last edited Tuesday
Hi @AnhPT ,
1 and 2. I’ll combine these two since it might be helpful—I can tell you that I’ve used this approach successfully. In a situation I’ve encountered with similar archetypes—where utilities or steps were exactly the same across bundles—and where, most importantly, if one changes, they all must change (cluster configurations, catalogs, run and manage permissions, etc.)—I’ve used a cross-bundle shared repository to store those utilities or core components used by the rest. This was accessed by defining the relative paths to it. This way, if I update a function or a notebook, when I deploy it to the next environment, every new job will use it. In my case, I deploy it using SPNs in a folder with read-only access for users, while the SPNs have execute permissions to compile it in the jobs—ensuring that “no one” can make manual changes.
3. Without a doubt, the best option for this—and to streamline the process (including agents)—is to use custom templates during initialization, whether from a Git/DevOps tool or directly within the workspaces. https://docs.databricks.com/aws/en/dev-tools/bundles/templates#custom-bundle-templates