Hi Everyone,
I am running job task using Asset Bundle.
Bundle has been validated and deployed according to: https://learn.microsoft.com/en-us/azure/databricks/dev-tools/bundles/work-tasks
Part of the databricks.yml
bundle:
name: etldatabricks
resources:
jobs:
etldatabricks-job:
name: etldatabricks-job
tasks:
- task_key: etldatabricks-python-script-task
existing_cluster_id: xxxx-17xx21-roxx
spark_python_task:
python_file: ./ingestion/my_script_dir/my_script.py
targets:
dev:
mode: development
default: true
I am receiving following error message during run of the my_script.py as a task in dev target via asset bundle.
NameError: name '__file__' is not defined
# my_script.py
import sys
from pathlib import Path
from databricks.connect import DatabricksSession
from pyspark.sql import DataFrameWriter
from pyspark.sql.types import StructType, StructField, StringType
# add root dir to path
root_dir_path = Path(__file__).parent.parent.parent
sys.path.append(str(root_dir_path))
I cannot understand why python dunder __file__ variable cannot be resolved running the script via DAB.
File of course works without any issues during standard databricks job run.
Thank you for any help.
Rafal