Databricks Community

kmodelew

I have a project witch my commons, like sparksession object (to run code in pycharm using databricks connect library and the same code directly on databricks).I have under src a few packages from which DAB creates separate jobs. I'm using PyCharm. S...

kmodelew · 03-26-2024

Hi,I would like to check if the task within job is succeded (even the job is marked as failed because on of the tasks).I need to create dependency for tasks within other jobs. The case is that I have one job for loading all tables for one country. Re...

kmodelew

Hi, You can use any of existing IDE. I'm using pycharm. I have created my own utils to run code on databricks. In .env file I have environmental variables and using SDK I'm creating SparkSession object and WorkspaceObject that you can use to read cre...

kmodelew

Hi, I hope it would be usefuel. Here are my files: project structure -> DAB_project_structure.pngeach yml file for job definitions -> task_group_1_job.png and task_group_2_job.pngEach .py file has main() method.setup.py:description="wheel file based ...

kmodelew

What are the permissions to this databricks directory? Can someone delete this directory or any file? On Shared workspace everyone can delete bundle files or bundle directory, even if in databricks.yml I provided permissions only to admins ('CAN MANA...

kmodelew · 03-26-2024

@Retired_mod Thank you. In my case I plan to use API:https://docs.databricks.com/api/workspace/jobs/listruns - for choosing the latest run_idhttps://docs.databricks.com/api/workspace/jobs/getrun - for getting task status for run_idPython script will ...

Databricks Community

User Stats

User Activity

Do I need many wheels for each job in project?

TaskSensor - check if task is succeded

Re: Local Development on Databricks

Re: Do I need many wheels for each job in project?

Re: How to copy notebooks from local to the tarrget folder via asset bundles

Re: TaskSensor - check if task is succeded