Can't get databricks 0.30.0 to load consistently.

chakan2 — Tue, 10 Sep 2024 15:18:44 GMT

Hi all,

I've been fighting with an issue for several weeks and it's driving me nuts. By default, databricks in our environment uses 0.20.0 of the databricks-sdk. I've been trying to update that to 0.30.0 using the pip magic work or via the requirements.txt file and installing it that way.

It looks like the package installs correctly. When I run 'version' on it, I get back 0.30.0...however, when I finally get around to calling my code...

`databricks_group = next(dbk_ac_client.groups.list(filter=f"displayName eq '{group}'"))`

I end up with this:
`Error was: 'list' object is not an iterator`

Seems like the lists changed to iterators sometime in the last few versions of the SDK.

Any help would be much appreciated!

Re: Can't get databricks 0.30.0 to load consistently.

chakan2 — Wed, 11 Sep 2024 15:30:58 GMT

I can't delete the original...but if someone runs into this...make sure your worker is using the latest version of spark. That's tied to the databricks-sdk version.

topic Can't get databricks 0.30.0 to load consistently. in Data Engineering

Can't get databricks 0.30.0 to load consistently.

Re: Can't get databricks 0.30.0 to load consistently.