Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
03-15-2024 07:19 AM
I just faced the same problem. The issue is that the when you do
JobSettings.as_dict()the settings are parsed to a dict where all the values are also parsed recursively. When you pass the parameters as **params, the create method again tries to parse each parameter to a dict since it needs to create a full json body to make the API request.
The trick is to parse only the first level of JobSettings to dict (not recursively) and create the params dict.
Try this:
import os
import time
from databricks.sdk import WorkspaceClient
from databricks.sdk.service import jobs
from databricks.sdk.service.jobs import JobSettings
w = WorkspaceClient()
notebook_path = f'/Users/{w.current_user.me().user_name}/sdk-{time.time_ns()}'
cluster_id = w.clusters.ensure_cluster_is_running(
os.environ["DATABRICKS_CLUSTER_ID"]) and os.environ["DATABRICKS_CLUSTER_ID"]
params = {
"name":f'sdk-{time.time_ns()}',
"tasks": [jobs.Task(description="test",
existing_cluster_id=cluster_id,
notebook_task=jobs.NotebookTask(notebook_path=notebook_path),
task_key="test",
timeout_seconds=0)],
'email_notifications': jobs.JobEmailNotifications(
no_alert_for_skipped_runs=True,
on_failure= ['some@email.com']
),
}
settings = JobSettings(
**params
)
# This only creates a dictionary but preserving the types of the values.
# Go to https://github.com/databricks/databricks-sdk-py/blob/33b209bdbaca4a956c7375fd65cbf8d6c3b4d95d/databricks/sdk/service/jobs.py and check that some settings has it's own classes (types) with its own method .to_dict()
create_params = {k:getattr(settings,k) for k in settings.as_dict().keys()}
created_job = w.jobs.create(**params) # this works
created_job = w.jobs.create(**settings.as_dict()) # this does not
created_job = w.jobs.create(**create_params) # this does