topic Re: Installing Bazel on databricks cluster in Data Engineering

Installing Bazel on databricks cluster

JGil — Wed, 12 Apr 2023 06:35:16 GMT

I am new to azure databricks and I want to install a library on a cluster and to do that I need to install bazel build tool first.

I checked the site bazel but I am still not sure how to do it in databricks?

I appriciate if any can help me and write me some steps

Re: Installing Bazel on databricks cluster

Avinash_94 — Fri, 14 Apr 2023 07:25:06 GMT

Databricks migrated over from the standard Scala Build Tool (SBT) to using Bazel to build, test and deploy our Scala code. Follow this doc https://www.databricks.com/blog/2019/02/27/speedy-scala-builds-with-bazel-at-databricks.html

Re: Installing Bazel on databricks cluster

JGil — Fri, 14 Apr 2023 07:41:47 GMT

Thanks, I read this article but I am still not sure how to make use of it. The original aim is to install library which explicitly mentioned the requirement to install first bazel. The article just explain how to use bazel without pointing how to install it or build it. Any more assistance , appriciate it

Re: Installing Bazel on databricks cluster

Avinash_94 — Fri, 14 Apr 2023 07:51:34 GMT

Hi Found this github repo talking about rules https://github.com/acqio/rules_databricks

Re: Installing Bazel on databricks cluster

JGil — Fri, 14 Apr 2023 08:39:57 GMT

Thanks, but isn't this repo also assume that bazel is installed?

I am saying this because of the second step in the link

"Add the flag --build_python_zip following to your bazelrc to create a python executable zip: "

so it assume i have bazelrc ! ?

Re: Installing Bazel on databricks cluster

Avinash_94 — Fri, 14 Apr 2023 09:13:40 GMT

I think we are looking at it wrong i think bezel need to be installed in a separate linux vm and then execute after configuring databricks cli on VM as it ask in authentication section. Post we are able to interact with databricks cluster via cli we can install libraries using this https://github.com/acqio/rules_databricks/blob/master/docs/dbk_libraries.md