How to speed up `dbx launch --from-assets`
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-09-2023 01:51 AM
Hiya,
I'm trying to follow the testing workflow of
```
$ dbx deploy test --assets-only
$ dbx launch test --from-assets --trace --include-output stdout
```
But I find the turnaround time is quite long, even with an instance pool.
The `deployment.yaml` looks like
```
environments:
default:
workflows:
- name: "test"
tasks:
- task_key: "main"
new_cluster:
spark_version: "11.3.x-scala2.12"
num_workers: 1
instance_pool_id: "instance-pool://****"
init_scripts:
- dbfs:
destination: dbfs://****/init.sh
python_wheel_task:
package_name: "foo"
entry_point: "bar"
parameters: [
"--conf-file", "file:fuse://conf/tasks/bar.yaml",
]
```
The `init.sh` simply does a `pip install`.
Some excerpts from the logs, to show how long things take
```
[dbx][2023-02-09 10:41:57.487] Launching workflow
[dbx][2023-02-09 10:42:01.389] Run URL: https://dbc-.
[dbx][2023-02-09 10:47:37.229] Finished tracing run with id ....
```
So it takes about 5 minutes setting up a new environment!
Anyone has ideas of how to speed up things?
- Labels:
-
Dbx
-
Instance Pool
-
Python
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-09-2023 10:54 PM
Oh no, do we get chat bots in here now?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-21-2023 07:23 AM
lol. maybe. But yes, if you're unclear about the many essay kinds, you might look at https://writinguniverse.com/essay-types/process-essays/ website to consider the several essay formats that may assist you in completing your essay writing work.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
02-21-2023 08:05 AM
Hi, I have no solution, actually I've just registered to open a very similar ticket, when saw yours.
According to my experiments getting an already running VM from the pool (times between events: CREATING - INIT_SCRIPTS_STARTED) can take anything between 5seconds and 5minutes. It's unclear why.
It's actually faster not to use the instance pools according to my experiments.
Additionally it would be great if at least the global ini scripts would be applied on the VMs when they are started in the pool (via that one could do some time consuming init steps independently of the job start).
Since you received no answer in a reltively long time, I guess one has to accept that it's just simply slow..... :'(

