cancel
Showing results for 
Search instead for 
Did you mean: 
Data Engineering
Join discussions on data engineering best practices, architectures, and optimization strategies within the Databricks Community. Exchange insights and solutions with fellow data engineers.
cancel
Showing results for 
Search instead for 
Did you mean: 

How to speed up `dbx launch --from-assets`

agagrins
New Contributor III

Hiya,

I'm trying to follow the testing workflow of

```

$ dbx deploy test --assets-only

$ dbx launch test --from-assets --trace --include-output stdout

```

But I find the turnaround time is quite long, even with an instance pool.

The `deployment.yaml` looks like

```

environments:

 default:

  workflows:

   - name: "test"

    tasks:

     - task_key: "main"

      new_cluster:

       spark_version: "11.3.x-scala2.12"

       num_workers: 1

       instance_pool_id: "instance-pool://****"

       init_scripts:

        - dbfs:

          destination: dbfs://****/init.sh

      python_wheel_task:

       package_name: "foo"

       entry_point: "bar"

       parameters: [

        "--conf-file", "file:fuse://conf/tasks/bar.yaml",

       ]

```

The `init.sh` simply does a `pip install`.

Some excerpts from the logs, to show how long things take

```

[dbx][2023-02-09 10:41:57.487] Launching workflow

[dbx][2023-02-09 10:42:01.389] Run URL: https://dbc-.

[dbx][2023-02-09 10:47:37.229] Finished tracing run with id ....

```

So it takes about 5 minutes setting up a new environment!

Anyone has ideas of how to speed up things?

3 REPLIES 3

agagrins
New Contributor III

Oh no, do we get chat bots in here now?

lol. maybe. But yes, if you're unclear about the many essay kinds, you might look at https://writinguniverse.com/essay-types/process-essays/ website to consider the several essay formats that may assist you in completing your essay writing work.

tonkol
New Contributor II

Hi, I have no solution, actually I've just registered to open a very similar ticket, when saw yours.

According to my experiments getting an already running VM from the pool (times between events: CREATING - INIT_SCRIPTS_STARTED) can take anything between 5seconds and 5minutes. It's unclear why.

It's actually faster not to use the instance pools according to my experiments.

Additionally it would be great if at least the global ini scripts would be applied on the VMs when they are started in the pool (via that one could do some time consuming init steps independently of the job start).

Since you received no answer in a reltively long time, I guess one has to accept that it's just simply slow..... :'(

Connect with Databricks Users in Your Area

Join a Regional User Group to connect with local Databricks users. Events will be happening in your city, and you won’t want to miss the chance to attend and share knowledge.

If there isn’t a group near you, start one and help create a community that brings people together.

Request a New Group