topic Re: Getting Spark & Scala version in Cluster node initialization script in Data Engineering

Getting Spark & Scala version in Cluster node initialization script

ahuarte — Thu, 16 Dec 2021 10:11:09 GMT

Hi there,

I am developing a Cluster node initialization script (https://docs.gcp.databricks.com/clusters/init-scripts.html#environment-variables) in order to install some custom libraries.

Reading the docs of Databricks we can get some environment variables with data related with the current running cluster node.

But I need to figure out what Spark & Scala version is currently been deployed. Is this possible?

Thanks in advance

Regards

Re: Getting Spark & Scala version in Cluster node initialization script

ahuarte — Thu, 16 Dec 2021 16:32:39 GMT

Hi Kaniz, thank you very much. For sure I will learn very much in this forum.

Re: Getting Spark & Scala version in Cluster node initialization script

Prabakar — Thu, 16 Dec 2021 18:12:10 GMT

Hi @A Huarte you can get the spark and scala version from the DBR that you will be using on the cluster.

Re: Getting Spark & Scala version in Cluster node initialization script

Prabakar — Thu, 16 Dec 2021 18:22:08 GMT

Re: Getting Spark & Scala version in Cluster node initialization script

ahuarte — Thu, 16 Dec 2021 18:35:31 GMT

Hi @Prabakar Ammeappin Thank you very much for your response,

but I mean how I can get this info in a script. I am trying to develop this sh init script for several Clusters with different Databricks runtimes.

I tried it searching files in that script but I did not find any "*spark*.jar" file from where to extract the current version of the runtime (Spark & Scala version).

When the cluster is already started there are files with this pattern, but in the moment that the init script is executed it seems that pyspark is not installed yet.

Re: Getting Spark & Scala version in Cluster node initialization script

ahuarte — Thu, 16 Dec 2021 18:37:29 GMT

I know that Databricks CLI tool is available, but it is not configured when the init script is running.

Re: Getting Spark & Scala version in Cluster node initialization script

sean_owen — Fri, 17 Dec 2021 02:16:59 GMT

Hm, this is a hacky idea, maybe there is a better way, but you could

ls /databricks/jars/spark*

and parse the results to get the version of Spark and Scala. You'll see files like spark--command--command-spark_3.1_2.12_deploy.jar containing the versions.

Re: Getting Spark & Scala version in Cluster node initialization script

ahuarte — Fri, 17 Dec 2021 10:23:26 GMT

Hi @Sean Owen thanks four your reply,

your idea can work, but unfortunatelly there is any filename with the full version name. I am missing the minor part:

yyyyyy_spark_3.2_2.12_xxxxx.jar -> Spark version is really 3.2.0

I have configured databricks CLI to get metadata of the cluster and I get this output:

{

"cluster_id": "XXXXXXXXX",

"spark_context_id": YYYYYYYYYYYY,

"cluster_name": "Devel - Geospatial",

"spark_version": "10.1.x-cpu-ml-scala2.12", ##<------!!!!

....

}

"spark_version" property does not contain info about the spark version but about the DBR :-(, any thoughts?

Thanks in advance

regards

Alvaro

Re: Getting Spark & Scala version in Cluster node initialization script

sean_owen — Fri, 17 Dec 2021 13:16:03 GMT

Do you need such specific Spark version info, why? should not matter for user applications

Re: Getting Spark & Scala version in Cluster node initialization script

ahuarte — Fri, 17 Dec 2021 13:31:05 GMT

I am trying to install Geomesa, from: https://mvnrepository.com/artifact/org.locationtech.geomesa/geomesa-gt-spark-runtime

from:

https://github.com/locationtech/geomesa/releases

I think I need the exact release.

Re: Getting Spark & Scala version in Cluster node initialization script

sean_owen — Fri, 17 Dec 2021 14:32:32 GMT

I doubt it's sensitive to a minor release, why?

But you also control what DBR/Spark version you launch the cluster with

Re: Getting Spark & Scala version in Cluster node initialization script

ahuarte — Fri, 17 Dec 2021 16:53:53 GMT

Many thanks @Sean Owen I am going to apply your advice, I am not going to write a generic init script that figures out everything, but a specific version of it for each Cluster type, really we only have 3 DBR types.

Thank you very much for your support

Regards

Re: Getting Spark & Scala version in Cluster node initialization script

Anonymous — Mon, 27 Dec 2021 17:17:18 GMT

@A Huarte - How did it go?

Re: Getting Spark & Scala version in Cluster node initialization script

ahuarte — Tue, 28 Dec 2021 11:44:19 GMT

Hi,

My idea was to deploy Geomesa or Rasterframes on Databricks in order to provide spatial capabilities to this platofrm. Finally, according to some advices in Rasterframes Gitter chat I selected the DBR 9.0 where I am installing pyrasterframes 0.10.0 via "pip" and no getting any errors.

I hope this info can be help.

Regards

Re: Getting Spark & Scala version in Cluster node initialization script

Anonymous — Fri, 31 Dec 2021 16:10:01 GMT

Thank you so much! Would you be happy to mark whichever answer is best in your mind? That will help new members know which is the most effective.

Re: Getting Spark & Scala version in Cluster node initialization script

User16764241763 — Thu, 27 Jan 2022 14:14:20 GMT

You should be able to just pick the version that matches Spark and Scala from maven.

Here is a simple way to get the cluster Spark version

Re: Getting Spark & Scala version in Cluster node initialization script

sean_owen — Thu, 27 Jan 2022 14:26:50 GMT

The question is about an init script though

Re: Getting Spark & Scala version in Cluster node initialization script

Lingesh — Wed, 14 Sep 2022 16:02:25 GMT

We can infer the cluster DBR version using the env $DATABRICKS_RUNTIME_VERSION. (For the exact spark/scala version mapping, you can refer to the specific DBR release notes)

Sample usage inside a init script,

DBR_10_4_VERSION="10.4"
if [[ "$DATABRICKS_RUNTIME_VERSION" == "$DBR_10_4_VERSION"* ]]; then
  echo "running 10.4 specific commands"
else
  echo "Skipping 10.4 specific commands"
fi