topic Re: Containerized Databricks/Spark database in Data Engineering

Containerized Databricks/Spark database

knight007 — Fri, 08 Apr 2022 08:33:32 GMT

Hello. I'm fairly new to Databricks and Spark.

I have a requirement to connect to Databricks using JDBC and that works perfectly using the driver I downloaded from the Databricks website ("com.simba.spark.jdbc.Driver")

What I would like to do now is have a locally running instance of a database in docker that I can connect to using the same driver. I'd like to automatically initialise the database by creating tables when it starts up. Very much like how you would use docker-entrypoint-initdb.d when creating tables on startup for Postgresql.

I'd then like to insert some data and run some tests locally.

Is any of this possible?

Re: Containerized Databricks/Spark database

-werners- — Fri, 08 Apr 2022 12:27:50 GMT

the simba driver is for spark connections, I doubt it will work with a database.

Why would you use this driver to connect to a database in a container? Or do you mean running Databricks in a local container?

If the latter: that is not available.

Re: Containerized Databricks/Spark database

knight007 — Fri, 08 Apr 2022 13:01:43 GMT

ok so maybe I've not asked the right question.

At the moment we use the Simba driver to connect to databricks and we can perform sql queries.

Can I achieve the same thing locally using a dockerized Databricks or spark runtime?

Re: Containerized Databricks/Spark database

-werners- — Fri, 08 Apr 2022 13:06:44 GMT

Databricks: no,

Spark: I guess so, but it will take some effort to gather all necessary dependencies and create a container (or look for one on dockerhub)

Re: Containerized Databricks/Spark database

-werners- — Fri, 08 Apr 2022 13:07:51 GMT

If you are just running queries on tables, you could also look into something like Dremio which can be run on docker in single node.

Re: Containerized Databricks/Spark database

knight007 — Fri, 08 Apr 2022 13:11:14 GMT

can I connect to that using the same Simba driver?

Re: Containerized Databricks/Spark database

-werners- — Fri, 08 Apr 2022 14:21:20 GMT

I don´t know if the Databricks driver is the same as the classic simba driver.

Re: Containerized Databricks/Spark database

Hubert-Dudek — Mon, 11 Apr 2022 20:29:20 GMT

@Gurps Bassi , "running instance of a database in docker" - that is hive metastore, so it just mapping to data which is usually physically on the data lake. Databricks are so much on the cloud that setting metastore locally doesn't make sense. Instead, place two Databricks workspaces, one with stage Repo (where you will make development) and another workspace with master Repo.

For those who want to develop locally in IDE, soon, the databricks tunnel for Visual Studio Code will be available.