Spark verison errors in "Build an ETL pipeline with Lakeflow Spark Declarative Pipelines"

liquibricks
Databricks Partner

‎12-29-2025 03:26 AM

I'm trying to define a job for a pipeline using the Asset Bundle Python SDK. I created the pipeline first (using the SDK) and i'm now trying to add the Job. The DAB validates and deploys successfully, but when I run the Job i get an error:

 

UNAUTHORIZED_ERROR: User <some-guid> does not have Run permissions on pipeline None.
 
How can I define the job to link to the already existing pipeline (which is already running in Continuous mode)?
 
The DAB code is as follows:
 
my_pipeline = Pipeline(
    name = "My Pipeline",
    catalog = "mycatalog",
    schema = "default",
    continuous=True,
    clusters = [
        PipelineCluster(
            ...
        )
    ],
    libraries = [
        PipelineLibrary(
            file=FileLibrary(path="src/my_sdp.py")
        )
    ]
)

my_task = Task(
    task_key="My_pipeline_task",
    pipeline_task=PipelineTask(
        pipeline_id=str(my_pipeline.id)
    )
)

my_job = Job(
    name="My Pipeline Job",
    tasks=[
        my_task
    ]
)
Labels: