Hi all,
I'm encountering some issues within my DLT pipelines. Summarized: it takes a long time to install the cluster libraries and dependencies (using %pip installs) due to horribly slow download speeds.
These are the symptoms:
- From all purpose clusters: 300-800 mb/s
- From job clusters: +- 300-800mb/s
- From dlt clusters: +- 5-25mb/s
These are the effects:
- A single iteration of developing my dlt pipeline takes 10-15 minutes because that's what it takes to get in the dependencies. This is not a workable development flow.
I have the following remarks:
- The download speed is consistent accross the dependencies, it does not seem it is particularly slow for certain dependencies.
- Yes, I could trim some of the dependencies dependent on the flow Im working as workaround. This is not desired, and downloading pyspark (+-300 MB) would be a hassle with these speeds.
- The infra in setup in an environment with private connectivity.
I'm trying to get a grasp whether this is usual behavior, and if not what the problem might be.
Does anyone have experience with something like this?
Please let me know what kind of information you would additionally like to help me out here.
