cancel
Showing results forย 
Search instead forย 
Did you mean:ย 
Data Engineering
cancel
Showing results forย 
Search instead forย 
Did you mean:ย 

How to speed up `dbx launch --from-assets`

agagrins
New Contributor III

Hiya,

I'm trying to follow the testing workflow of

```

$ dbx deploy test --assets-only

$ dbx launch test --from-assets --trace --include-output stdout

```

But I find the turnaround time is quite long, even with an instance pool.

The `deployment.yaml` looks like

```

environments:

 default:

  workflows:

   - name: "test"

    tasks:

     - task_key: "main"

      new_cluster:

       spark_version: "11.3.x-scala2.12"

       num_workers: 1

       instance_pool_id: "instance-pool://****"

       init_scripts:

        - dbfs:

          destination: dbfs://****/init.sh

      python_wheel_task:

       package_name: "foo"

       entry_point: "bar"

       parameters: [

        "--conf-file", "file:fuse://conf/tasks/bar.yaml",

       ]

```

The `init.sh` simply does a `pip install`.

Some excerpts from the logs, to show how long things take

```

[dbx][2023-02-09 10:41:57.487] Launching workflow

[dbx][2023-02-09 10:42:01.389] Run URL: https://dbc-.

[dbx][2023-02-09 10:47:37.229] Finished tracing run with id ....

```

So it takes about 5 minutes setting up a new environment!

Anyone has ideas of how to speed up things?

3 REPLIES 3

agagrins
New Contributor III

Oh no, do we get chat bots in here now?

lol. maybe. But yes, if you're unclear about the many essay kinds, you might look at https://writinguniverse.com/essay-types/process-essays/ website to consider the several essay formats that may assist you in completing your essay writing work.

tonkol
New Contributor II

Hi, I have no solution, actually I've just registered to open a very similar ticket, when saw yours.

According to my experiments getting an already running VM from the pool (times between events: CREATING - INIT_SCRIPTS_STARTED) can take anything between 5seconds and 5minutes. It's unclear why.

It's actually faster not to use the instance pools according to my experiments.

Additionally it would be great if at least the global ini scripts would be applied on the VMs when they are started in the pool (via that one could do some time consuming init steps independently of the job start).

Since you received no answer in a reltively long time, I guess one has to accept that it's just simply slow..... :'(

Welcome to Databricks Community: Lets learn, network and celebrate together

Join our fast-growing data practitioner and expert community of 80K+ members, ready to discover, help and collaborate together while making meaningful connections. 

Click here to register and join today! 

Engage in exciting technical discussions, join a group with your peers and meet our Featured Members.