topic Re: How to speed up `dbx launch --from-assets` in Data Engineering

How to speed up `dbx launch --from-assets`

agagrins — Thu, 09 Feb 2023 09:51:32 GMT

Hiya,

I'm trying to follow the testing workflow of

```

$ dbx deploy test --assets-only

$ dbx launch test --from-assets --trace --include-output stdout

```

But I find the turnaround time is quite long, even with an instance pool.

The `deployment.yaml` looks like

```

environments:

default:

workflows:

- name: "test"

tasks:

- task_key: "main"

new_cluster:

spark_version: "11.3.x-scala2.12"

num_workers: 1

instance_pool_id: "instance-pool://****"

init_scripts:

- dbfs:

destination: dbfs://****/init.sh

python_wheel_task:

package_name: "foo"

entry_point: "bar"

parameters: [

"--conf-file", "file:fuse://conf/tasks/bar.yaml",

]

```

The `init.sh` simply does a `pip install`.

Some excerpts from the logs, to show how long things take

```

[dbx][2023-02-09 10:41:57.487] Launching workflow

[dbx][2023-02-09 10:42:01.389] Run URL: https://dbc-.

[dbx][2023-02-09 10:47:37.229] Finished tracing run with id ....

```

So it takes about 5 minutes setting up a new environment!

Anyone has ideas of how to speed up things?

Re: How to speed up `dbx launch --from-assets`

agagrins — Fri, 10 Feb 2023 06:54:36 GMT

Oh no, do we get chat bots in here now?

Re: How to speed up `dbx launch --from-assets`

ErinArmistead — Tue, 21 Feb 2023 15:23:54 GMT

lol. maybe. But yes, if you're unclear about the many essay kinds, you might look at https://writinguniverse.com/essay-types/process-essays/ website to consider the several essay formats that may assist you in completing your essay writing work.

Re: How to speed up `dbx launch --from-assets`

tonkol — Tue, 21 Feb 2023 16:05:07 GMT

Hi, I have no solution, actually I've just registered to open a very similar ticket, when saw yours.

According to my experiments getting an already running VM from the pool (times between events: CREATING - INIT_SCRIPTS_STARTED) can take anything between 5seconds and 5minutes. It's unclear why.

It's actually faster not to use the instance pools according to my experiments.

Additionally it would be great if at least the global ini scripts would be applied on the VMs when they are started in the pool (via that one could do some time consuming init steps independently of the job start).

Since you received no answer in a reltively long time, I guess one has to accept that it's just simply slow..... :'(