Databricks

philnally · ‎04-05-2023

I am looking for direction and input: I am an SAP data architect, with around 20 years of expossure to the data model itself, plus I’ve installed the SAP Data Warehouse some 6, or 7 times. I am certified in SAP’s newest analytics Tool, SAP Analytics Cloud. And separately I’ve created a library full of SQL scripts to access and display various clients’ SAP data needs. I see the world from within the monastery.

Ant here is what I am contemplating – I would like to jump into the Databricks pool, and aim for ‘Databricks Engineer’ certification.

I see SAP as a data machine, and databricks as an engineering machine, so with the recent Datasphere + Databricks connection (not an engagement yet) I see a great potential, so some constructive criticism would be appreciated.

Just to show I’m not completely green, I have been playing with Python on and off for several years, and I also have Jupyter Notebook set up a couple of years ago with some 20 projects. Separately, I have some Machine Learning, mainly time series, understanding.

Am I crazy? I’d appreciate any comments, or input. Is there a perception that the SAP + Databricks connection is great? What do you think?

Anonymous · ‎04-06-2023

@Michael McNally :

Based on your experience and knowledge, I am thrilled to let you know that you have a strong foundation in data architecture and analytics within the SAP ecosystem. Moving towards Databricks and pursuing a Databricks Engineer certification could be a good way to expand your skillset and explore new opportunities.

Databricks is a popular platform for big data processing and analytics, and there is a growing demand for engineers with experience in Databricks. Your experience with Python and Jupyter Notebook will be a valuable asset as well. The learning curve wont be steep.

-werners- · ‎04-06-2023

Hi,

I did the same thing years ago. At the time (10 years ago) I saw the potential of big data and distributed compute (like Spark) and decided to focus on something else than what SAP delivered. It was not easy as you can imagine, coming from a closed eco system like SAP.

But nowadays things have evolved rapidly and I see no reason why it should not be possible.

The biggest mindshift is not learning to program in python/scala/... but 'thinking distributed'.

Learning where Spark excels and where not, getting used to lazy evaluation, learning functional programming concepts (which I believe is overlooked very often, it is so valuable though) etc.

SAP + Databricks integration is certainly valuable as SAP tends to be pretty closed (even if they claim not to be). Databricks just wants to get data, so if SAP opens up, that would be pretty great. Of course, the proof of the pudding... SAP has promised a lot in the past.

So anyhow: do it, you are already late 🙂