https://community.databricks.com/t5/technical-blog/from-idea-to-code-building-with-the-databricks-sdk-for-python/ba-p/58894 <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-right" image-alt="Databricks Python SDK launch" style="width: 400px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/6058iEF430D45F50C4705/image-size/medium?v=v2&amp;px=400" role="button" title="kimberlyma_0-1706723334148.png" alt="Databricks Python SDK launch" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Databricks Python SDK launch</span></span>Six months ago Databricks announced the release of the Databricks SDK for Python to much fanfare. Since then it has been adopted by over 1,000 customers and is used in several open source tools such as </SPAN><A href="https://github.com/datahub-project/datahub" target="_blank" rel="noopener"><SPAN>Datahub</SPAN></A><SPAN>.</SPAN></P> <P><SPAN>Over the past six months I've worked with many folks - helping answer questions or creating bespoke code snippets for their projects. We have robust </SPAN><A href="https://docs.databricks.com/en/dev-tools/sdk-python.html" target="_blank" rel="noopener"><SPAN>documentation</SPAN></A><SPAN> for the SDK as well as a repository of </SPAN><A href="https://github.com/databricks/databricks-sdk-py#code-examples" target="_blank" rel="noopener"><SPAN>examples</SPAN></A><SPAN>, but, there are not examples for every single thing you may want to do with the SDK. So, I figured it will benefit some folks to walk through how I approached familiarizing myself with the codebase and how to develop new projects quickly.&nbsp;</SPAN></P> <P><SPAN>The focus of this blog is to demystify the Python SDK - the authentication process and the components of the SDK by walking through the start-to-end development process. I'll also be showing how to utilize IntelliSense and the Debugger for real-time suggestions in order to reduce the amount of context-switching from the IDE to documentation and code examples.</SPAN></P> <H2><SPAN>What is the Databricks SDK for Python</SPAN></H2> <P><I><SPAN>And why you should use it</SPAN></I></P> <P><SPAN>The Databricks Python SDK lets you interact with the Databricks Platform programmatically using Python. It covers the entire Databricks API surface and Databricks REST operations. While you can interact directly with the API via curl or a library like '</SPAN><SPAN><FONT face="courier new,courier">requests</FONT>'</SPAN><SPAN>&nbsp;there are benefits to utilizing the SDKs such as:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>Secure and simplified authentication via </SPAN><A href="https://docs.databricks.com/en/dev-tools/auth/index.html#databricks-client-unified-authentication" target="_blank" rel="noopener"><SPAN>Databricks client-unified authentication</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Built-in debug logging with sensitive information automatically redacted</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Support to wait for long-running operations to finish (kicking off a job, starting a cluster)</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Standard iterators for paginated APIs (we have multiple pagination types in our APIs!)</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>Retrying on transient errors</SPAN></LI> </UL> <P><SPAN>There are numerous practical applications, such as building multi-tenant web applications that interact with your ML models or a robust UC migration toolkit like Databricks Labs project</SPAN><A href="https://github.com/databrickslabs/ucx" target="_blank" rel="noopener"> <SPAN>UCX</SPAN></A><SPAN>. Don’t forget the silent workhorses; those simple utility scripts that are more limited in scope but automate an annoying task such as bulk updating cluster policies, dynamically adding users to groups, or simply writing data files to UC Volumes. Implementing these types of scripts is a great way to familiarize yourself with the Python SDK and Databricks APIs.</SPAN></P> <H2><SPAN>Scenario</SPAN></H2> <P><SPAN>Imagine my business is establishing best practices for development and CI/CD on Databricks. We're adopting </SPAN><A href="https://docs.databricks.com/en/dev-tools/bundles/index.html" target="_blank" rel="noopener"><SPAN>DABs</SPAN></A><SPAN> to help us define and deploy workflows in our development and production Workspaces, but in the meantime, we need to audit and clean up our current environments. We have a lot of jobs people created in our dev Workspace via the UI. One of the platform admins observed many of these jobs are inadvertently configured to run on a recurring schedule, racking up unintended costs. As part of the clean-up process, we want to identify any scheduled jobs in our development Workspace with an option to pause them. We’ll need to figure out:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><SPAN>How to install the SDK</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>How to connect to the Databricks Workspace</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>How to list all the jobs and examine their attributes</SPAN></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>How to log the problematic jobs - or a step further, how to call the API to pause their schedule</SPAN></LI> </UL> <H2><SPAN>Development environment</SPAN></H2> <P><SPAN>Before diving into the code, you need to set up your development environment. I highly recommend using an IDE that has a comprehensive code completion feature as well as a debugger. Code completion features, such as</SPAN><A href="https://code.visualstudio.com/docs/editor/intellisense" target="_blank" rel="noopener"> <SPAN>Intellisense in VS Code</SPAN></A><SPAN>, are really helpful when learning new libraries or APIs - they provide useful contextual information, autocompletion, and aid in code navigation. For this blog, I’ll be using Visual Studio Code so I can also make use of the</SPAN><A href="https://docs.databricks.com/en/dev-tools/vscode-ext/index.html" target="_blank" rel="noopener"> <SPAN>Databricks Extension</SPAN></A><SPAN> as well as</SPAN><A href="https://marketplace.visualstudio.com/items?itemName=ms-python.vscode-pylance" target="_blank" rel="noopener"> <SPAN>Pylance</SPAN></A><SPAN>. You’ll also need to install the databricks-sdk (</SPAN><A href="https://docs.databricks.com/en/dev-tools/sdk-python.html#get-started-with-the-databricks-sdk-for-python&amp;language-Poetry" target="_blank" rel="noopener"><SPAN>docs</SPAN></A><SPAN>). In this blog, I’m using Poetry + Pyenv. The setup is similar for other tools - just '</SPAN><FONT face="courier new,courier">poetry add databricks-sdk'</FONT><SPAN> or alternatively</SPAN><FONT face="courier new,courier">&nbsp;'pip install databricks-sdk'</FONT><SPAN> in your environment.</SPAN></P> <H2><SPAN>Authentication</SPAN></H2> <P><SPAN>The next step is to authorize access to Databricks so we can work with our Workspace. There are several ways to do this, but because I’m using the VS Code Extension I’ll take advantage of its authentication integration. It’s one of the tools that uses </SPAN><A href="https://docs.databricks.com/en/dev-tools/auth/index.html#databricks-client-unified-authentication" target="_blank" rel="noopener"><SPAN>unified client authentication</SPAN></A><SPAN> - that just means all these development tools follow the same process and standards for authentication and if you set up auth for one you can reuse it amongst the other tools. I set up both the CLI and VS Code Extension previously, but here is a primer on </SPAN><A href="https://docs.databricks.com/en/dev-tools/cli/install.html" target="_blank" rel="noopener"><SPAN>setting up the CLI </SPAN></A><SPAN>and </SPAN><A href="https://docs.databricks.com/en/dev-tools/vscode-ext/install.html" target="_blank" rel="noopener"><SPAN>installing the extension.</SPAN></A><SPAN> Once you’ve connected successfully you’ll see a notification banner in the lower right-hand corner and see two hidden files generated in the&nbsp;</SPAN><I><SPAN>.databricks</SPAN></I><SPAN> folder - </SPAN><I><SPAN>project.json</SPAN></I><SPAN> and </SPAN><I><SPAN>databricks.env</SPAN></I><SPAN> (don’t worry, the extension also handles adding these to .gitignore).&nbsp;</SPAN></P> <P><SPAN>For this example, while we’re interactively developing in our IDE we’ll be using what’s called U2M (user-to-machine) OAuth. We won’t get into the technical details, but OAuth is a secure protocol that handles authorization to resources without passing sensitive user credentials such as PAT or username/password that persist much longer than the one-hour short-lived OAuth token.</SPAN></P> <P class="lia-align-center"><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="OAuth flow for the Databricks Python SDK" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/6070iA0042C33AD0BBB8F/image-size/large?v=v2&amp;px=999" role="button" title="image7.gif" alt="OAuth flow for the Databricks Python SDK" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">OAuth flow for the Databricks Python SDK</span></span></P> <H2><SPAN>WorkspaceClient vs. AccountClient</SPAN></H2> <P><SPAN>The Databricks API is split into two primary categories - Account and Workspace. They let you manage different parts of Databricks, like user access at the account level or cluster policies in a Workspace. The SDK reflects this with two clients that act as our entry points to the SDK - the <FONT face="courier new,courier">WorkspaceClient</FONT> and <FONT face="courier new,courier">AccountClient</FONT>. For our example we’ll be working at the Workspace level so I’ll be initializing the <FONT face="courier new,courier">WorkspaceClient</FONT>. If you're unsure which client to use, check out the </SPAN><A href="https://databricks-sdk-py.readthedocs.io/en/latest/clients.html" target="_blank" rel="noopener"><SPAN>SDK documentation</SPAN></A><SPAN>. </SPAN></P> <P><SPAN><span class="lia-unicode-emoji" title=":light_bulb:">💡</span>Because we ran the previous steps to authorize access&nbsp;via unified client auth, the SDK will automatically use the necessary Databricks environment variables, so there's no need for extra configurations when setting up your client. All we need are these two lines of code:</SPAN></P> <UL> <LI><EM>Initializing our WorkspaceClient</EM><LI-CODE lang="python">from databricks.sdk import WorkspaceClient w = WorkspaceClient()</LI-CODE></LI> </UL> <H2><SPAN>Making API calls and interacting with data&nbsp;</SPAN></H2> <P><SPAN>The Workspace client we instantiated will allow us to interact with different APIs across the Databricks Workspace services. A </SPAN><STRONG>service</STRONG><SPAN> is a smaller component of the Databricks Platform i.e. Jobs, Compute, Model Registry. In our example, we’ll need to call the Jobs API in order to retrieve a list of all the jobs in the Workspace.&nbsp;</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-inline" image-alt="Services accessible via the Python SDK" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/6065i797064EA27742E20/image-size/large?v=v2&amp;px=999" role="button" title="image1.png" alt="Services accessible via the Python SDK" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Services accessible via the Python SDK</span></span></P> <P><SPAN>This is where Intellisense really comes in handy. Instead of context switching between the IDE and the Documentation page I can use autocomplete to provide a list of methods as well as examine the method description, the parameters, and return types from within the IDE. I know the first step is getting a list of all the jobs in the Workspace:</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="image6.png" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/6066i793334FB97CEC8B5/image-size/large?v=v2&amp;px=999" role="button" title="image6.png" alt="image6.png" /></span><BR /><SPAN>As you can see it returns an iterator over an object called BaseJob. Before we talk about what a BaseJob actually is it’ll be helpful to understand how data is used in the SDK. To interact with data you are sending to and receiving from the API, the Python SDK takes advantage of Python <A href="https://docs.python.org/3/library/dataclasses.html" target="_self"><STRONG>Data Classes</STRONG></A> and <A href="https://docs.python.org/3/library/enum.html" target="_self"><STRONG>Enums</STRONG></A>. The main advantage of this approach over passing around dictionaries is improved readability while also minimizing errors through enforced type checks and validations.</SPAN></P> <P><SPAN>You can construct objects with Data Classes and interact with enums. For example:</SPAN></P> <UL> <LI style="font-weight: 400;" aria-level="1"><I><SPAN>Creating an Employee via Employee DataClass and company departments, using Enums for possible department values</SPAN></I><I></I><LI-CODE lang="python">from dataclasses import dataclass from enum import Enum from typing import Optional class CompanyDepartment(Enum): MARKETING = 'MARKETING' SALES = 'SALES' ENGINEERING = 'ENGINEERING' @dataclass class Employee: name: str email: str department: Optional[CompanyDepartment] = None emp = Employee('Bob', 'bob@company.com', CompanyDepartment.ENGINEERING)</LI-CODE></LI> </UL> <P><SPAN>In the Python SDK all of the data classes, enums, and APIs belong to the same module for a service located under</SPAN> <STRONG>databricks.sdk.service</STRONG><SPAN> - e.g. <FONT face="courier new,courier">databricks.sdk.service.<STRONG>jobs</STRONG>, databricks.sdk.service.<STRONG>billing</STRONG>, databricks.sdk.service.<STRONG>sql</STRONG>.<BR /><BR /></FONT></SPAN><SPAN>For our example, we'll need to loop through all of the jobs and make a decision on whether or not they should be paused. I'll be using a </SPAN><A href="https://code.visualstudio.com/docs/editor/debugging" target="_blank" rel="noopener"><SPAN>debugger</SPAN></A><SPAN> in order to look at a few example jobs and get a better understanding of what a ‘<FONT face="courier new,courier">BaseJob</FONT>’ looks like. The Databricks VS Code extension comes with a debugger that can be used to troubleshoot code issues interactively on Databricks via</SPAN><A href="https://docs.databricks.com/en/dev-tools/vscode-ext/dev-tasks/databricks-connect.html" target="_blank" rel="noopener"> <SPAN>Databricks Connect</SPAN></A><SPAN>. But, because I do not need to run my code on a cluster I'll just be using the standard&nbsp;</SPAN><A href="https://code.visualstudio.com/docs/python/debugging" target="_blank" rel="noopener">Python debugger</A><SPAN>. I’ll set a breakpoint inside my for loop and use the VS Code Debugger to examine a few examples. A breakpoint allows us to stop code execution and interact with variables during our debugging session. This is preferable over print statements, as you can use the debugging console to interact with the data as well as progress the loop. In this example I’m looking at the </SPAN><STRONG>settings</STRONG><SPAN> field and drilling down further in the debugging console to take a look at what an example job schedule looks like:</SPAN></P> <P><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Inspecting BaseJob in the VS Code debugger" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/6067iCA37E83B73FB65AC/image-size/large?v=v2&amp;px=999" role="button" title="image5.png" alt="Inspecting BaseJob in the VS Code debugger" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Inspecting BaseJob in the VS Code debugger</span></span></P> <P><SPAN>We can see a <FONT face="courier new,courier">BaseJob</FONT> has a few top-level attributes and has a more complex <FONT face="courier new,courier">Settings</FONT> type that contains most of the information we care about.&nbsp;</SPAN><SPAN>At this point, we have our Workspace client and are iterating over the jobs in our Workspace. To flag problematic jobs and potentially take some action we’ll need to better understand </SPAN><STRONG>job.settings.schedule</STRONG><SPAN>. We need to figure out how to programmatically identify if a job has a schedule and flag if it’s not paused. For this we’ll be using another handy utility for code navigation - </SPAN><A href="https://code.visualstudio.com/docs/editor/editingevolved#_go-to-definition" target="_blank" rel="noopener"><SPAN>Go to Definition</SPAN></A><SPAN>. I’ve opted to Open Definition to the Side (⌘K F12) in order to reduce switching to a new window. This will allow us to quickly navigate through the DataClass definitions without having to switch to a new window or exit our IDE:</SPAN></P> <P class="lia-align-center"><IFRAME src="https://www.youtube.com/embed/C5UQviJg93c?si=R1eZEtJHWtjpnq7m" width="760" height="515" frameborder="0" allowfullscreen="" title="YouTube video player" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share"></IFRAME></P> <P><SPAN>As we can see, a <FONT face="courier new,courier">BaseJob</FONT> contains some top-level fields that are common amongst Jobs such as 'job_id' or 'created_time'. A job can also have various settings (<FONT face="courier new,courier">JobSettings</FONT>). These configurations often differ between Jobs and encompass aspects like notification settings, tasks, tags, and the schedule. We’ll be focusing on the schedule field, which is represented by the <FONT face="courier new,courier">CronSchedule</FONT> data class. <FONT face="courier new,courier">CronSchedule</FONT> contains information about the pause status (<FONT face="courier new,courier">PauseStatus</FONT>) of a job. <FONT face="courier new,courier">PauseStatus</FONT> in the SDK is represented as an enum with two possible values - <STRONG>PAUSED</STRONG> and <STRONG>UNPAUSED</STRONG>.&nbsp;</SPAN></P> <P class="lia-indent-padding-left-30px"><STRONG><span class="lia-unicode-emoji" title=":light_bulb:">💡</span></STRONG><STRONG>Tip</STRONG><SPAN>: </SPAN><SPAN>VSCode + Pylance provides code suggestions and you can enable auto imports in your </SPAN><A href="https://code.visualstudio.com/docs/getstarted/settings" target="_blank" rel="noopener"><SPAN>User Settings or on a per-project basis in Workspace Settings</SPAN></A><SPAN>.</SPAN> <SPAN>By default, only top-level symbols are suggested for auto import and suggested code (</SPAN><A href="https://github.com/microsoft/pylance-release/issues/2533#issuecomment-1089268739" target="_blank" rel="noopener"><SPAN>see original GitHub issue</SPAN></A><SPAN>). However, the SDK has nested elements we want to generate suggestions for. We actually need to go down 5 levels - databricks.sdk.service.jobs.&lt;Enum|Dataclass&gt;. In order to take full advantage of these features for the SDK I added a couple of Workspace settings:</SPAN></P> <UL> <LI><SPAN><I>Selection of theVSCode Workspace settings.json</I></SPAN><LI-CODE lang="javascript">... "python.analysis.autoImportCompletions": true, "python.analysis.indexing": true, "python.analysis.packageIndexDepths": [ { "name": "databricks", "depth": 5, "includeAllSymbols": true } ] ... </LI-CODE></LI> </UL> <H3>Putting it all together:</H3> <P><SPAN>I broke out the policy logic into its own function for unit testing, added some logging, and expanded the example to check for any jobs tagged as an exception to our policy. Now we have:</SPAN></P> <UL> <LI><I><SPAN>Logging out of policy jobs</SPAN></I><I></I><LI-CODE lang="python">import logging from databricks.sdk import WorkspaceClient from databricks.sdk.service.jobs import CronSchedule, JobSettings, PauseStatus # Initialize WorkspaceClient w = WorkspaceClient() def update_new_settings(job_id, quarts_cron_expression, timezone_id): """Update out of policy job schedules to be paused""" new_schedule = CronSchedule( quartz_cron_expression=quarts_cron_expression, timezone_id=timezone_id, pause_status=PauseStatus.PAUSED, ) new_settings = JobSettings(schedule=new_schedule) logging.info(f"Job id: {job_id}, new_settings: {new_settings}") w.jobs.update(job_id, new_settings=new_settings) def out_of_policy(job_settings: JobSettings): """Check if a job is out of policy. If it unpaused and has a schedule and is not tagged as keep_alive Return true if out of policy, false if in policy """ tagged = bool(job_settings.tags) proper_tags = tagged and "keep_alive" in job_settings.tags paused = job_settings.schedule.pause_status is PauseStatus.PAUSED return not paused and not proper_tags all_jobs = w.jobs.list() for job in all_jobs: job_id = job.job_id if job.settings.schedule and out_of_policy(job.settings): schedule = job.settings.schedule logging.info( f"Job name: {job.settings.name}, Job id: {job_id}, creator: {job.creator_user_name}, schedule: {schedule}" ) ....​</LI-CODE></LI> </UL> <P><SPAN>Now we have not only a working example but also a great foundation for building out a generalized job monitoring tool. We're successfully connecting to our Workspace, listing all the jobs, and analyzing their settings, and, when we're ready, we can simply call our `</SPAN><FONT face="andale mono,times">update_new_settings function</FONT><SPAN>` to apply the new paused schedule. It's fairly straightforward to expand this to meet other requirements you may want to set for a Workspace - for example, swap job owners to service principles, add tags, edit notifications, or audit job permissions.&nbsp;See the example in the<A href="https://github.com/kimberlyma/dev-job-monitoring" target="_blank" rel="noopener"> GitHub repository</A>.</SPAN></P> <H2><SPAN>Scheduling a job on Databricks</SPAN></H2> <P><SPAN>You can run your script anywhere, but you may want to schedule scripts that use the SDK to run as a Databricks Workflow or job on a small single-node cluster. When running a Python notebook interactively or via automated workflow you can take advantage of default Databricks Notebook authentication. If you're working with the Databricks Workspace client and your cluster meets the </SPAN><A href="https://docs.databricks.com/en/dev-tools/sdk-python.html#use-the-databricks-sdk-for-python-from-a-databricks-notebook" target="_blank" rel="noopener"><SPAN>requirements listed in the docs</SPAN></A><SPAN> you can initialize your WorkspaceClient without needing to specify any other configuration options or environment variables - it works automatically out of the box.</SPAN></P> <P><SPAN><span class="lia-inline-image-display-wrapper lia-image-align-center" image-alt="Using the Python SDK in a job" style="width: 999px;"><img src="https://community.databricks.com/t5/image/serverpage/image-id/6069i883C72E5E7C7A1E5/image-size/large?v=v2&amp;px=999" role="button" title="image4.png" alt="Using the Python SDK in a job" /><span class="lia-inline-image-caption" onclick="event.preventDefault();">Using the Python SDK in a job</span></span></SPAN></P> <H2><SPAN>Conclusion</SPAN></H2> <P><SPAN>In conclusion, the Databricks SDKs offer limitless potential for a variety of applications. We saw how the Databricks SDK for Python can be used to automate a simple yet crucial maintenance task and also saw an example of an OSS project that uses the Python SDK to integrate with the Databricks Platform. Regardless of the application you want to build, the SDKs streamline development for the Databricks Platform and allow you to focus on your particular use case. The key to quickly mastering a new SDK such as the Databricks Python SDK is setting up a proper development environment. Developing in an IDE allows you to take advantage of features such as a debugger, parameter info, and code completion, so you can quickly navigate and familiarize yourself with the codebase. Visual Studio Code is a great choice for this as it provides the above capabilities and you can utilize the VSCode extension for Databricks to benefit from unified authentication.</SPAN></P> <P><SPAN>Any feedback is greatly appreciated and welcome. Please raise any issues in the</SPAN><A href="https://github.com/databricks/databricks-sdk-py" target="_blank" rel="noopener"> <SPAN>Python SDK GitHub repository</SPAN></A><SPAN>.&nbsp;&nbsp;Happy developing! </SPAN></P> <H3><SPAN>Additional resources:</SPAN></H3> <UL> <LI style="font-weight: 400;" aria-level="1"><A href="https://databricks-sdk-py.readthedocs.io/en/latest/" target="_self"><SPAN>Databricks SDK for Python Documentation</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>DAIS Presentation: </SPAN><A href="https://www.youtube.com/watch?v=HWKJV_cRT9M" target="_blank" rel="noopener"><SPAN>Unlocking the Power of Databricks SDKs</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><SPAN>How to install Python libraries in your local development environment:&nbsp;</SPAN><A href="https://www.youtube.com/watch?v=0f3moPe_bhk" target="_blank" rel="noopener"><SPAN>How to Create and Use Virtual Environments in Python With Poetry</SPAN></A></LI> <LI style="font-weight: 400;" aria-level="1"><A href="https://docs.databricks.com/en/dev-tools/vscode-ext/install.html" target="_blank" rel="noopener"><SPAN>Installing the Databricks extension for Visual Studio Code</SPAN></A></LI> </UL> Thu, 01 Feb 2024 19:25:48 GMT kimberlyma 2024-02-01T19:25:48Z https://community.databricks.com/t5/technical-blog/from-idea-to-code-building-with-the-databricks-sdk-for-python/ba-p/58894 <P><SPAN>This blog unpacks the components of the Databricks SDK for Python and gives helpful tips along the way in order to accelerate your development cycle. We'll walk through a simple, but powerful, real-world example of bulk-auditing and updating jobs in a Databricks workspace.</SPAN></P> Thu, 01 Feb 2024 19:25:48 GMT https://community.databricks.com/t5/technical-blog/from-idea-to-code-building-with-the-databricks-sdk-for-python/ba-p/58894 kimberlyma 2024-02-01T19:25:48Z https://community.databricks.com/t5/technical-blog/from-idea-to-code-building-with-the-databricks-sdk-for-python/bc-p/88827#M265 <P>Hi,</P><P>I’m looking to create a dependency between jobs. Specifically, I need to add a task in Job2 that checks the recent run status of Job1, as Job2 depends on Job1. Is there a function (similar to an external job sensor) or a specific method to achieve this?</P><P>Thanks in advance!</P> Fri, 06 Sep 2024 05:36:02 GMT https://community.databricks.com/t5/technical-blog/from-idea-to-code-building-with-the-databricks-sdk-for-python/bc-p/88827#M265 mohan_2724 2024-09-06T05:36:02Z https://community.databricks.com/t5/technical-blog/from-idea-to-code-building-with-the-databricks-sdk-for-python/bc-p/90808#M279 <P>Hi&nbsp;<a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/119518">@mohan_2724</a>&nbsp;. Have you already tried the Run Job task type? We do have a Workflows-native way to do this using a Python interface along with some other trigger types. Feel free to e-mail me <A href="mailto:kimberly.mahoney@databricks.com" target="_blank">kimberly.mahoney@databricks.com</A>&nbsp;to discuss further and I can help loop in the correct folks.&nbsp;</P> Wed, 18 Sep 2024 03:19:23 GMT https://community.databricks.com/t5/technical-blog/from-idea-to-code-building-with-the-databricks-sdk-for-python/bc-p/90808#M279 kimberlyma 2024-09-18T03:19:23Z https://community.databricks.com/t5/technical-blog/from-idea-to-code-building-with-the-databricks-sdk-for-python/bc-p/90814#M280 <P>Hi <a href="https://community.databricks.com/t5/user/viewprofilepage/user-id/28632">@kimberlyma</a>&nbsp;,</P><P>Thanks for your response.</P><P>We can't use Run Job Task Type, because those are not sequential and they scheduled for different timings. Please help me with any other approch.</P><P>Thanks &amp; Regards,</P><P><SPAN>Mohan&nbsp;</SPAN></P> Wed, 18 Sep 2024 03:58:59 GMT https://community.databricks.com/t5/technical-blog/from-idea-to-code-building-with-the-databricks-sdk-for-python/bc-p/90814#M280 mohan_2724 2024-09-18T03:58:59Z