How to extract data from SAP to Databricks?

[RaulTovar]

Hey everyone, I'm working through a project where I need to get data out of SAP S4Hana and into Databricks, and I'd love to pick the brains of anyone who's done this before.

A few things I'm trying to figure out:

• How are you actually pulling the data out? CDS views, ODP extractors, some third party connector, or a custom build?
• Are you going real time (Kafka/CDC style) or is scheduled batch working fine for your use case?
• What's bitten you the most? Schema changes breaking things, SAP getting cranky under load, latency, something else entirely?

We're dealing with decent volume on our end and really don't want to hammer the SAP source system, so reliability matters more to us than raw speed. If you've been through this, I'd really appreciate hearing what worked (and what didn't).

Thanks in advance!

[Sumit_7]

Hi @RaulTovar,
Please go the below thread. Hopeful you'll get your questions answered.
https://community.databricks.com/t5/data-engineering/data-migration-from-sap-s-4hana-to-databricks/t...

Also, feel free to check/search on the community as someone might have asked the same question before.

Thanks!

[jerrygen78]

Thanks Sumit this helps me 

[syedshahul]

Hi,
SAP BDC is one option to consider.

SAP Databricks in Business Data Cloud 
sap-bdc OpenSharing 

[dima]

@RaulTovar, a big chunk of the questions I have answered in a sibling thread, but to address the missing part:

• You go with the style you actually need, but most likely you'll end up with batch extraction using a delta mechanism. This can be built into the extractor using CDC-related annotations in an ABAP CDS view that serves as an extractor. Just make sure the specific view supports the required delta mechanism.

• SAP is pretty robust internally (too robust for some), so schema evolution on the SAP side is usually manageable, but changes to CDS views, custom fields, extractor interfaces and downstream schemas can still break things.

What I think is important here:

1. Realise that a CDS view is rarely built per table in SAP; it is usually already a chunk of logic in itself.

2. These CDS views are executed virtually when the extractor reads them, meaning that joins and other logic can run on the source system and compete for its limited resources.

3. Many SAP customers are large international businesses operating 24/7 across multiple geographies, so good batch scheduling is essential. You need to consider the load on the source system, not just the latency you want on the Databricks side.

4. Latency depends on several things: the extraction and delta mechanism, the complexity of the CDS view, source-system load, the middleware pulling the data and moving it to storage, and the network in between.

I'd do a PoC with a properly scoped, realistic data volume. One candidate to test is C_SalesDocumentItemDEX_1. It is an SAP-delivered extraction-enabled CDS view for sales document items, so it can give you a feel for the performance and extraction behaviour.

Hope this helps.

Cheers,
Dmitry Kuznetsov

[anksaini]

Easiest would be to use SAP BDC