topic Databricks and DDD in Administration & Architecture

Databricks and DDD

Dunken — Tue, 18 Mar 2025 16:50:06 GMT

Our architecture is according to Domain Driven Design. The data is therefore distributed among different domains.

We would like to run workloads on top of our data but we would like to avoid to have a dedicated (duplicated) data lake just for Databricks. Instead we would rather like to directly rely on our own data sources (accessible via REST APIs) in order to always run on the same, latest data.

Could anybody point me to some resources in order to get started? It would definitively be fine to have an abstraction layer between what we use in a notebook and how our backend APIs look like...

Re: Databricks and DDD

Hubert-Dudek — Thu, 27 Jan 2022 17:17:44 GMT

You can just use urlopen or requests and than read json as a dataframe using spark.json(). Problem is that in that case you will need handle whole logic (when to load data, how to handle incremental load etc).

Easier solution o is to use streaming and put kafka with data from your API ( confluent.io can be registered also through Azure) or any other stream like eventHubs and than your newest data can be read as kafka stream in databricks and processed data will be saved in destination of your choice. Maybe on side of your infrastructure you can just deploy microservice which read from rest apis and save to stream.

Re: Databricks and DDD

-werners- — Fri, 28 Jan 2022 07:13:00 GMT

So basically you do not want to persist data outside of your source systems.

I think the so called 'Kappa architecture' could be a fit, where everything is treated like a stream.

Hubert already mentioned Kafka, which is an excellent source to build this (there are also others). And on top of that you could use Spark, or Flink or whatever.

There is also Apache Nifi and Streamsets and ...

Kappa architecture is pretty cool, but not without it's flaws.

There is also the pretty recent 'Data mesh', where providing data is seen as domain-based. This could be a match for your use case.

But this approach of course also has it's flaws (governance, gigantic overhead f.e.)

Re: Databricks and DDD

Dunken — Fri, 28 Jan 2022 16:51:12 GMT

Thanks. If I would use streaming I would replicate all my data sources, isn't it? This is actually something I would like to avoid... also, because I don't know up-front in which data I'm interested in I would have to store everything at Databricks.

Re: Databricks and DDD

-werners- — Mon, 31 Jan 2022 13:15:40 GMT

if you really want to avoid replicating data (so this means reporting directly on your source systems), you can look into Presto or Trino or Dremio etc.