Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-06-2025 11:02 AM - edited 01-06-2025 11:04 AM
Hello,
This is the yaml file I used, but btw I think because I cannot use a cluster up and never stop, I had the problem that during every new deploy, btw the cluster retains the first deploy... seems weird.
resources:
jobs:
sync_tables_gold_with_pg:
name: sync_tables_gold_with_pg
tasks:
- task_key: sync_tables_gl_and_pg
job_cluster_key: job_new_cluster
existing_cluster_id: 1224-151003-c62b2avz
notebook_task:
notebook_path: ../notebook/sync_gold_tables_to_postgres.ipynb
source: WORKSPACE
base_parameters:
env: ${bundle.target}
libraries:
- whl: ${workspace.root_path}/files/dist/pipelines-0.0.1-py3-none-any.whl